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OFFICE  OF  THE  SECRETARY  OF  DEFENSE 
WASHINGTON,  DC  20301-1700 


OPERATIONAL 
TEST  AND 
EVALUATION 

SUBJECT:  OPERATIONAL  SUITABILITY  MODELING  AND  SIMULATION 

Operational  suitability  modeling  and  simulation  (M&S)  should  n&  be  viewed  as  a  substitute  for  field 
testing.  Our  primary  objective  is  to  acquire  maximum  operational  suitability  data  from  field  testing 
with  representative  equipment,  typical  operators,  and  typical  support  personnel,  in  a  stressful  opera¬ 
tional  environment. 

This  report  on  the  application  of  modeling  and  simulation  to  operational  suitability  as  an  adjunct  to 
actual  testing  is  important  for  many  reasons: 

‘"  TSst;  if  is  'published  to  provide  guidance,  examples,  and  reference  materials  to  improve  the 
consistency  of  M&S  application.  The  report  is  intended  to  be  a  beginning~not  an  end--and  to  be  a 
living  document  that  will  continue  to  evolve  and  provide  cuijent  and  helpful  information  towards 
improved  operational  suitability  OT&E.  C  .  -  -  J  ^ 


_  SecondTTF  fepfesents  a  clear  statement  of  the  importance  of  operational  suitability  and  the  need  to 
highlight  and  improve  its  assessment  during  OT&E.  Clearly,  the  1980’s  were  the  decade  of 
recognition  of  the  importance  and  impact  of  operational  suitability.  This  document  will  help  DoD  in 
its  quest  for  greatly  improved  operational  suitability  of  our  weapon  systems. 

Third7iL^vfdes  a  heeded  forum--a  forum  to  clarify  the  proper  role  of  M&S  in  OT&E.,  As  with  any 
tool,  M&S  can  be  misused,  even  over  used.  Some  basic  ground  rules  are: 

•  A  primary  role  for  operational  suitability  M&S  is  the  development  and  analysis  \ 
of  OT&E  parameters,  hypotheses,  and  criteria.  As  a  result,  there  will  be  a 
better  understanding  of  the  relevance  of  data  collected  during  test  and  the 
criticality  of  attaining  specified  operational  suitability  levels. 

•  Under  closely  controlled  conditions,  operational  suitability  M&S  can  povide  a 
structure  for  assessments  that  cannot  be  attained  through  field  testing.  For 
example,  the  pace  and  scale  of  operations  during  most  field  tests  do  not  provide 
an  environment  for  the  direct  measurement  of  system  operational  availability. 

M&S  can  provide  insights  on  the  impact  of  test  results  on  these  measures. 

•  DOT&E  understands  the  potential  for  misuse  of  M&S  and  will  work  with  the 
Oversight  Agencies  and  the  Operational  Test  Agencies  to  ensure  that  the 
application  of  M&S  is  properly  planned,  controlled,  and  documented  and  that 
the  models  and  simulations  themselves  are  poperly  verified  and  validated. 


In  summary,  I  support  and  encourage  tin  application  of  M&S  as  a  valuable  tool  that  can  greatly 
enhance  the  value  of  OT&E  to  the  deck  makers.  At  the  same  time,  we  must  all  work  together  to 
ensure  that  our  application  of  M&S  will  stand  the  common  sense  tests  of  credibility,  intelligence,  and 
integrity  and  that  it  can  never  displace  the  need  for  actual  testing  under  real  operational  conditions. 


Robert  C.  Duncan 
Director 


FOREWORD 


The  effective  operational  test  and  evaluation  (OT&E)  of  defense  systems  is  a  critical  part  of  the 
on-going  program  to  provide  for  the  proper  defense  for  the  United  States.  The  Department  of 
Defense  has  an  established  process  for  planning  and  conducting  operational  tests  and  for 
evaluating  the  data  that  result  from  those  tests. 

The  role  of  modeling  and  simulation  in  this  OT&E  activity  has  grown  in  importance  in  recent 
years.  Modeling  and  simulation  (M&S)  can  be  important  supplements  to  the  actual  operational 
testing  of  new  defense  systems.  In  the  operational  suitability  area,  M&S  can  be  even  more 
important.  Through  M&S,  the  critical  suitability  aspects  of  a  system  and  the  compatibility  of  the 
weapon  with  its  support  system  can  be  examined.  This  report  was  prepared  to  highlight  the 
reasons  why  M&S  is  used  in  the  operational  suitability  area  of  OT&E  and  to  describe  some  of 
the  key  process  steps  that  are  needed  for  effective  application  of  M&S. 

It  should  be  noted  that  the  application  of  M&S  also  will  vary  with  size,  scope,  and  complexity  of 
the  system,  and  the  acquisition  phase  of  the  system.  Some  tools  and  techniques  are  scalable  from 
small  systems  to  very  large  systems.  For  small  systems,  spreadsheets  or  small  simulation  models 
may  suffice.  In  some  cases,  these  same  tools  may  be  of  great  value  for  very  large  systems. 
There  also  are  tools  and  techniques  that  are  very  useful  from  the  early  concept  development 
phase  well  into  production  and  fielding.  While  the  focus  of  this  document  is  on  larger  systems, 
the  concepts,  processes,  and  many  of  the  models  and  techniques  apply  across  all  systems. 

The  report  is  organized  in  the  following  manner.  Chapter  1  discusses  the  importance  of 
operational  suitability,  and  some  of  the  reasons  why  M&S  is  needed  for  effective  operational 
evaluation  of  most  defense  systems.  Chapter  2  discusses  the  use  of  M&S  in  the  operational 
suitability  area.  Chapter  3  describes  the  role  of  M&S  and  provides  an  outline  for  a  generic 
approach  to  operational  suitability  M&S.  Chapter  4  contains  example  applications  of  M&S  to 
the  OT&E  of  systems.  Chapter  5  provides  a  summary  of  the  information  that  is  in  the  report. 
The  Appendix  comprises  summary  information  on  12  models  that  have  been  used  in  the 
operational  suitability  area.  Summary  charts  that  provide  a  composite  view  of  all  the  areas 
covered  by  the  12  models  and  that  indicate  the  applicability  of  each  model  to  the  respective  suit¬ 
ability  areas  also  are  included  in  the  Appendix. 

If  questions  or  comments  arise  while  reviewing  or  using  this  document,  they  should  be 
forwarded  to  the  primary  author: 


Dr.  Elizabeth  Rodriguez 
The  Office  of  the  Director  of  Operational  Test 
and  Evaluation 

Office  of  the  Secretary  of  Defense 
The  Pentagon,  Room  1C730 
Washington,  DC  20301 

(202)  697-3895  J 


STATEMENT  "A"  per  Dr.  Rodriguez 
OSD/DOT&E  The  Pentagon,  Washington,  DC 
20301-1700 
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Chapter  1 
INTRODUCTION 


The  importance  of  Operational  Suitability  (OS)  is  no  longer  in  question.  Dozens  of  Department 
of  Defense  reports,  contractor  studies,  case  histories,  and  results  from  combat  operations  provide 
conclusive  evidence  that  operational  suitability  factors  are  not  only  critical  to  the  mission 
performance  of  weapons  systems,  but  also  easily  account  for  more  than  half  of  total  weapon 
system  costs.  Table  1-1  provides  a  list  of  some  of  the  DoD  efforts  directed  toward  evaluating 
operational  suitability,  and  the  efforts  undertaken  to  improve  the  suitability  of  new  weapon 
systems. 


Table  1-1. 

OPERATIONAL  SUITABILITY 
STUDIES  AND  IMPROVEMENT  EFFORTS 


EFFORT 

IMPACT 

DATE 

PUBLISHED 

Navy  "New  Look" 

Reliability  Program 

Major  Overhaul  of  Navy 

Approach  to  Reliability 

1975 

Carlucci  Study  on  Improving 

The  Acquisition  Process 

Actions  in  Incentives.  Funding, 
and  Acquisition  Management 
for  R&M  and  Support 

1981 

Report  of  The  Defense  Science 
Board  1981  Summer  Panel  on 
Operational  Readiness  with 

High  Performance  Systems 

Recommendations  for  Improving 
Reliability  Standards  for 

Operational  Availability,  Spares 
Purchase  and  Distribution, 

Visibility  of  System-by-System 
Readiness,  and  for  the  Establishment 
of  OSD-Level  Readiness  Advocate 

1982 

IDA/OSD  R&M  Study 

An  Analysis  of  New  Technology  and 
Potential  Contributions  to  R&M 
Problems  in  Their  Application, 
as  well  as  Impact  on  Achievement 
of  Desired  R&M  Goals 

1983 

U  S.  Air  Force  R&M  2000 

Initiative 

The  Establishment  of  Goals, 

Principles,  and  Building  Blocks 
for  Improving  Combat  Capability 
Through  R&M 

1985 

Manpower,  spare  parts,  support  equipment,  technical  data,  training,  sustaining  engineering,  and 
software  maintenance  are  major  direct  cost  elements,  both  during  the  acquisition  process  and 
throughout  the  life  cycle  of  the  system.  In  addition,  some  cost  effects  of  the  operational 
suitability  elements  show  up  only  when  in  the  aggregate.  An  example  is  the  need  for  more 
manpower,  or  other  suitability  elements,  due  to  shortfalls  in  one  or  more  of  the  suitability  areas 
such  as  reliability,  m  intainability,  or  support  equipment.  Critical  effects  also  can  be  felt  in 
terms  of  system  downtime,  lost  or  ineffective  missions,  lost  or  destroyed  equipment,  or  injury  to 
personnel. 
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1.1  Operational  Suitability  Continues  To  Be  a  Problem 

Despite  the  widespread  appreciation  of  its  critical  role,  incorporation  of  operational  suitability 
ractors  continues  to  present  significant  problems  in  most  weapon  system  development  programs. 
Given  the  clear  evidence  that  excellence  in  OS  is  a  necessary  attribute  of  successful  systems, 
why  is  it  still  a  problem  on  many  systems?  There  are  at  least  two  categories  of  factors  that 
contribute  to  this  situation.  The  first  relates  to  the  nature  of  suitability  factors  themselves  and  the 
second  concerns  the  manner  in  which  suitability  is  viewed  by  the  development  managers. 

1.1.1  Operational  Suitability  Factors 

The  nature  of  suitability  factors  themselves  —  their  essence  makes  it  difficult  to  manage  and 
control  operational  suitability  during  system  development.  For  example,  one  of  the  key 
obstacles  is  the  inability  to  directly  and  quantitatively  assess  reliability  and  maintainability 
(R&M),  or  supportability  elements,  early  in  the  development  process  because  data  required  to  do 
this  type  of  assessment  come  late  in  the  development  process.  Yet,  if  managers  wait  until  hard, 
tangible  results  are  available  before  they  assess  an  operational  suitability  product,  such  as 
technical  data  or  support  equipment,  an  opportunity  will  have  been  missed.  They  will  be  unable 
to  identify  and  correct  problems  before  the  expenditure  of  significant  amounts  of  program 
funding.  Thus,  program  managers  must  include  provisions  for  monitoring  and  controlling 
suitability  progress  within  their  program  management  tracking  systems. 

The  second  set  of  factors  —  the  manner  in  which  suitability  is  viewed  —  is  distinctly  different 
than  the  first,  although  its  manifestation  is  rooted  therein.  Since  suitability  elements  are  among 
the  final  pieces  of  a  system  to  be  completed,  funding  for  their  development  usually  is  still  "on  the 
table”  when  the  system  gets  into  trouble.  Thus,  the  funding  for  the  suitability  elements  is 
available  and  vulnerable  for  reallocation. 

1.1.2  The  Calculated  Risk 

To  those  who  stand  outside  the  system  development  process,  the  fact  that  systems  are  developed 
and  delivered  with  serious  suitability  problems  seems  bizarre.  Why  would  a  company  begin 
delivery  of  a  newly  designed  television,  automobile,  software  program,  tank,  ship,  or  aircraft 
when  the  support  elements  are  missing,  or  incomplete?  In  fact,  this  calculated  risk  often  is  taken. 
Most  people  have  experienced  or  have  known  someone  who  has  experienced  buying  a  new 
automobile,  having  a  problem,  and  finding  that  the  dealer  does  not  yet  have  the  equipment  to  test 
the  system,  or  the  mechanics  trained  to  fix  the  problem.  This  is  even  more  likely  to  happen  with 
a  new  piece  of  software. 

Why  is  the  calculated  risk  taken?  Why  are  products  delivered  with  incomplete  support  or  low 
quality  suitability  attributes?  It  is,  in  effect,  because  the  customer  needs  or  wants  something 
badly  enough  that  he/she  will  tolerate  potential  problems,  and  their  effects. 

There  also  are  situations  within  the  marketplace  that  stimulate  the  taking  of  calculated  risks.  For 
example,  with  a  new,  hot  item  that  other  competitors  are  close  to  delivering,  managers  can  be 
lead  into  a  decision  to  deliver  without  complete  testing  or  without  full  support.  The  environment 
within  which  defense  systems  are  developed  also  can  make  these  calculated  risks  attractive;  for 
example,  when  development  money  is  depleted  and  further  support  from  the  Services  or 
Congress,  without  deliveries,  is  not  likely.  Another  possible  stimulus  is  the  situation  where  too 
many  new  or  advanced  systems  are  competing  for  development  within  total  available  funding. 
The  assignment  of  unrealistically  low  funding  to  a  program  almost  always  has  a  negative  impact 
on  the  quality  of  the  suitability  elements  being  developed  for  that  program. 
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Accordingly,  as  long  as  the  conventional  wisdom  says  that  this  calculated  risk  is  a  good  risk,  i.e., 
experience  supports  the  action,  there  will  be  examples  of  where  the  risk  will  be  taken.  It 
probably  will  be  financially  rewarding  for  the  company,  and  the  customer  will  receive  a  system 
that  is  not  perfect,  but  is  better  than  the  one  it  replaces. 

This  type  of  dilemma  is  not  unusual  and  provides  one  of  the  compelling  reasons  for  strong 
analytical  (including  M&S)  support  for  OS.  The  ability  to  provide  timely  quantitative 
INSIGHTS  and  IMPACT  statements  is  absolutely  essential. 

1.2  The  Operational  Suitability  Challenge  Must  Be  Met 

One  of  the  greatest  challenges  in  the  operational  suitability  arena  is  to  be  able  to  assess  the 
potential  impact  (or  cost)  of  the  risk  from  the  customers'  perspectives.  The  objective  is  to 
quantify  the  risk  in  a  clear  and  concise  way,  and  in  terms  that  are  understandable  to  the 
decisionmaker.  Most  importantly,  this  quantification  and  assessment  must  be  completed  within 
the  timeframe  of  the  decisionmaker's  commitment  to  buy. 

The  most  desirable  approach  is  actual  testing!  --  testing  of  representative  equipment,  employed 
in  its  intended  environment,  operated  and  maintained  by  representative  military  personnel,  and 
operated  at  a  pace  and  scale  that  is  representative  of  its  intended  use.  In  numerous  cases,  these 
conditions  cannot  be  attained,  yet  the  decisionmakers  require  quantitative  inputs. 

This  challenge  for  quantification  provides  the  motivation  for  using  modeling  and  simulation 
(M&S)  in  the  area  of  operational  suitability.  M&S  can  provide  the  structure  and  the  process  to 

•  identify  and  assemble  the  required,  detailed  information, 

•  store  the  data  in  a  structured  way, 

•  facilitate  dialogue  among  contractors,  program 
management  personnel,  users,  and  testers, 

•  provide  a  mechanism  for  exercising  the  assembled  data 
structure  in  various  scenarios, 

•  contribute  to  the  database  for  trade  studies,  and 

•  provide  an  assessment  of  the  impact  of  suitability 
shortfalls. 

1.3  Special  Problems  Associated  with  Operational  Suitability  Testing 

The  test  and  evaluation  (T&E)  community  has  a  specific  requirement  to  assess  the  suitability  of 
each  system.  Meeting  this  requirement  presents  some  significant  problems  because  there  are 
many  limitations  to  testing  capability  in  the  OS  area  including  funding,  the  availability  of 
tangible  assets,  the  realism  of  the  available  test  assets,  the  number  of  test  assets  available,  the 
numbers  and  realism  of  the  test  events,  and  the  incorporation  of  evolving  technologies.  The 
following  sections  discuss  issues  associated  with  these  limitations. 
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1.3.1  Front-End  Funding  for  Suitability  Items 


There  is  a  tendency  to  delay  or  underfund  suitability  elements  in  the  early  phases  of  acquisition. 
The  reason  often  given  is  that  the  suitability  products  are  not  required  until  later.  The  work 
breakdown  structure  (WBS)  may  even  support  this  rationale  because  of  its  inherent  weakness  in 
stressing  that  the  core  material  for  input  to  the  suitability  products  be  developed  early  in  the 
acquisition,  and  be  captured  and  identified  as  such. 

1.3.2  Availability  of  Tangible  Assets 

During  the  early  phases  of  acquisition  and  prior  to  Initial  Operational  Test  and  Evaluation 
(IOT&E),  typical  test  assets  are  conceptual  models,  early  prototypes,  brassboards,  and  hot 
mockups.  These  assets  are  developed  for  the  purpose  of  proving  design  concepts  and  demonstrat¬ 
ing  feasibility,  as  well  as  other  objectives  that  are  specifically  aimed  at  Operational  Effectiveness 
(OE)  issues.  The  situation  is  not  significantly  different  at  the  point  of  IOT&E.  Often,  the 
logistics  support  infrastructure  is  not  sufficiently  developed  prior  to  a  Limited  Rate  or  a  Beyond 
Limited  Rate  production  decision  to  completely  evaluate  all  operational  suitability  areas.  Table 
1-2  summarizes,  for  a  nominal  system,  an  assessment  of  the  availability  of  suitability  test  assets 
at  the  acquisition  milestones. 

Table  1-2. 

AVAILABILITY  OF  OPERATIONAL  SUITABILITY  TEST  ASSETS 


OS  TEST  ASSETS 

MS-0 

MS-1 

MS-11 

MS- 1 11 A 

MS-I1IB 

Hardware 

Nz 

Nz 

Nz 

Min 

Sat 

Software 

Nz 

Nz 

Nz 

Nz 

Min 

Technical  Data 

Nz 

Nz 

Nz 

Nz 

Min 

Diagnostic  Systems 

Nz 

Nz 

Nz 

Nz 

Min 

Support  Equipment 

Nz 

Nz 

Nz 

Nz 

Min 

Training  Material 

Nz 

Nz 

Nz 

Nz 

Min 

Spare  Parts 

Nz 

Nz 

Nz 

Nz 

Nz 

Service  Personnel 

Nz 

Nz 

Nz 

Min 

Sat 

LEGEND:  Nz  -  Near  zero 
Min  -  Minimum 
Sat  =  Satisfactory 


1.3.3  Test  Asset  Fidelity 

Table  1-2  above  attempts  only  to  show  "availability"  of  the  OS  assets  that  would  be  required  to 
run  an  operational  test  of  sufficient  scope  to  produce  credible  results.  A  further  problem  is  that, 
even  on  full-scale  development  models,  many  of  the  assets  that  are  available  are  not  "production 
or  production  equivalent."  Thus,  suitability  testing  or  assessment  data  must  be  carefully 
examined  and  analyzed  before  presenting  any  results. 
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1.3.4  Sample  Size 

The  sample  size  issue  goes  further  than  the  number  and  fidelity  of  test  assets  discussed  above. 
Most  development  and  operational  test  designs  are  structured  to  provide  sufficient  replications  to 
measure  Operational  Effectiveness  (OE)  objectives.  Even  with  this  focus,  the  OE  replications 
are  not  always  achieved  because  of  the  limits  on  the  cost  of  testing.  In  nearly  every  case,  the 
limited  number  of  test  hours  provides  a  less  than  ideal  confidence  level  in  most  of  the  operational 
suitability  results. 

1.3.5  Test  Scenarios 

Test  scenarios  are  nearly  always  driven  by  OE  requirements  and  objectives.  Again,  this  is  not  to 
imply  that  achievable  levels  of  testing  always  are  sufficient  for  OE  objectives  and  that  merely  the 
testing  for  operational  suitability  is  limited.  In  fact,  it  is  seldom  cost  effective  to  approach  the 
projected  combat  pace  during  OT&E,  yet  many  of  the  logistics  measures  of  interest  are  best 
measured  at  or  near  the  expected  pace  of  combat.  How  do  we  replicate  the  surge  of  a  combat 
unit  with  only  a  few  test  articles? 

There  are  other  scenario  issues  as  well.  Should  both  peacetime  and  wartime  operating  scenarios 
be  examined  during  the  testing?  How  realistic  should  the  test  environment  be  in  comparison 
with  the  intended  operational  environment?  Is  it  possible  to  evaluate  the  suitability 
characteristics  in  an  intended,  realistic  operational  environment  (including  weather,  shock, 
vibration,  dust,  water,  battle  damage,  etc.)?  In  nearly  all  cases,  there  is  an  inability  to  use  or  test 
the  total  interface  to  the  planned  support  system. 

1.3.6  New  Technology 

Another  significant  challenge  is  the  incorporation  of  constantly  evolving  new  technology  into 
military  systems.  Driven  by  the  need  to  stay  ahead  of  the  evolving  threat,  there  is  a  constant 
search  for  advantage  through  the  application  of  the  latest  technology  to  defense  systems.  The 
impacts  on  OS  and  OE  testing  are  similar.  Advanced  technology  systems  result  in  an  on-going 
need  to  examine  test  methods,  and  to  develop  and  refine  them  accordingly. 
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Chapter  2 

THE  ROLE  OF  OPERATIONAL  SUITABILITY  M&S 


Modeling  and  simulation  can  play  a  significant  role  during  test  and  evaluation  for  operational 
suitability.  As  discussed  in  section  1.2,  M&S  can  provide  the  structure  and  the  process  to  (a) 
identify  and  assemble  the  required,  detailed  information,  (b)  store  the  data  in  a  structured  way, 

(c)  facilitate  dialogue  between  contractors,  program  management  personnel,  users,  and  testers, 

(d)  provide  a  mechanism  for  exercising  the  assembled  data  structure  in  various  scenarios,  (e) 
contribute  to  the  database  for  trade  studies,  and  (f)  provide  an  assessment  of  the  impact  of 
suitability  shortfalls. 

The  essential  utility  of  the  model  is  its  value  in  dialogue  and  communications.  As  can  be  seen 
above,  there  is  no  case  where  M&S  for  suitability  is  not  required.  Even  in  the  best  case  -  full 
recognition  of  the  criticality  of  suitability  and  support  for  suitability  funding  —  there  are 
shortfalls.  The  operational  suitability  asset  availability  still  has  voids,  and  issues  such  as  asset 
fidelity,  test  scenarios  (e.g.,  pace  of  combat),  and  interface  to  the  support  system  can  be  reason¬ 
ably  easy  to  handle  in  a  model. 

2.1  Definitions  of  Modeling  and  Simulation 

A  model  is  defined  in  DoD  5000.3-M-l  as  "...  a  representation  of  an  actual  or  conceptual  system 
that  involves  mathematics,  logical  expressions,  or  computer  simulations  that  can  be  used  to 
predict  how  the  system  might  perform  or  survive  under  various  conditions  or  in  a  range  of  hostile 
environments." 

Simulation  is  defined  in  DoD  5000.3-M-l  as  "...  a  method  for  implementing  a  model.  It  is  the 
process  of  conducting  experiments  with  a  model  for  the  purpose  of  understanding  the  behavior  of 
the  system  modeled  under  selected  conditions  or  of  evaluating  various  strategies  for  the 
operation  of  the  system  within  the  limits  imposed  by  developmental  or  operational  criteria." 
There  are  different  types  of  simulations,  including  those  that  use  analog  or  digital  devices, 
laboratory  models,  or  "testbed"  sites.  Simulations,  in  the  broadest  sense,  also  can  include 
military  exercises  and  wargames. 

2.2  Use  of  Suitability  M&S 

The  use  of  properly  validated  M&S  is  strongly  encouraged  during  the  early  phases  of  a  program 
to  assess  those  areas  that  cannot  be  directly  observed  through  testing.  The  use  of  M&S  is  not  a 
substitute  for  actual  testing;  however,  it  can  provide  early  projections  and  reduce  test  costs  by 
supplementing  actual  test  data.  The  DoD  is  in  the  process  of  issuing  expanded  guidance  on  the 
development,  validation,  and  use  of  M&S  in  the  acquisition  process.  In  January  1989,  the 
Director  of  Operational  Test  and  Evaluation  (DOT&E)  issued  "DOT&E  Policy  for  the 
Application  of  Modeling  and  Simulation  in  Support  of  Operational  Test  and  Evaluation." 

The  use  of  modeling  and  simulation  in  the  operational  suitability  area  can  provide  a  significant 
number  of  benefits.  For  example,  M&S  can  be  used  to  focus  limited  test  resources  by 
identifying  critical  elements  in  a  logistics  support  system,  e.g.,  the  choke  points  for  the  flow  of 
the  support  resources.  M&S  can  be  used  to  translate  the  rate  of  use  in  the  test  scenario  to  the 
wartime  usage  rate.  If  test  aircraft  are  flying  only  one  or  two  sorties  per  day,  the  "load"  on  the 
support  resources  is  significantly  different  than  if  a  higher,  wartime  sortie  rate  were  being  flown. 
M&S  can  aid  in  assessing  the  impact  of  these  differences.  M&S  also  can  be  used  to  evaluate 
elements  of  the  support  system  that  are  not  present  at  the  test  site.  If  the  second-level 
maintenance  capability,  e.g.,  test  equipment,  facilities,  etc.,  is  not  available,  then  a  properly 
constructed  and  validated  model  can  be  used  to  provide  insight  into  the  ability  of  the  planned 
second-level  maintenance  facility  to  support  the  system. 
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The  Appendix  to  this  report  contains  summary  descriptions  of  12  M&S  approaches  that  are  being 
used  or  developed  for  application  to  operational  suitability.  These  summaries  indicate  the 
application  area  of  each  item,  and  provide  a  short  description  of  the  capabilities  provided. 

2.3  Considerations  in  Suitability  M&S 

M&S  plans  should  be  evaluated  to  ensure  credibility  of  the  results.  Such  creditability  derives 
from  a  composite  impression  of  the  assumptions,  inputs,  processes,  outputs,  conclusions,  persons 
or  agencies  involved,  and  the  strength  of  the  evidence  presented.  Appendix  B  of  the  "DOT&E 
Policy  for  the  Application  of  Modeling  and  Simulation  in  Support  of  Operational  Test  and 
Evaluation"  provides  a  series  of  questions  to  assist  in  addressing  the  credibility  of  M&S  results. 
These  questions  provide  a  good  outline  for  examining  M&S  activities. 

Detailed  definitions  of  planned  operating  and  support  scenarios  are  essential  for  a  valid  M&S 
effort.  In  many  cases,  a  detailed  definition  beyond  that  in  program  documentation  is  needed  for 
M&S.  This  is  particularly  true  for  the  concepts  that  pertain  to  the  maintenance  and  supply 
activities  supporting  the  system.  This  requirement  for  expanded  detail  can  have  disadvantages  as 
well  as  advantages.  For  example,  it  is  difficult  to  apply  M&S  to  a  situation  that  is  not  defined  in 
adequate  detail;  there  is  potential  for  the  results  to  be  driven  by  some  of  the  necessary 
assumptions  rather  than  by  the  system  characteristics;  and  the  persons  conducting  the  modeling 
may  end  up  being  a  greater  part  of  the  solution  than  desired,  and  the  model  results  subsequently 
may  be  challenged.  On  the  other  hand,  if  the  responsible  personnel  and  organizations  provide 
the  required  definition  and  detail,  then  a  portion  of  the  detailed  support  planning  will  be 
completed  and  will  have  a  benefit  to  others  within  the  acquisition  program.  The  key  point  is  — 
what  assumptions  were  made  in  structuring  the  model  and  were  those  assumptions  realistic  in 
everyone's  mind. 

Furthermore,  the  latest  program  information  must  be  incorporated  into  an  M&S  activity.  A 
common  fault  of  many  modeling  efforts  is  this  lack  of  most  recent  information.  Program 
conditions  may  change.  The  system  design  may  be  revised,  or  new  threat  information  received. 
In  each  case,  the  earlier  model  may  be  invalidated.  Assuring  that  the  modeling  results  reflect  the 
best  information  available  is  an  important  consideration.  Procedures  need  to  be  established  to 
assure  that  current  information  is  provided  to  those  performing  the  modeling  and  evaluating  the 
results. 

2.4  Where  M&S  Fits 

In  many  situations,  there  is  no  substitute  for  the  real  thing.  In  the  test  and  evaluation  arena,  this 
means  "hands-on"  testing  by  appropriate  Service  personnel,  using  the  real  product,  in  the 
intended  operational  environment,  at  the  pace  of  battle,  and  with  the  intended  support  structure. 
Reality  has  it  differently,  however.  Cost,  time,  political  considerations,  and  other  factors  must  be 
constantly  reviewed  and  balanced.  Any  move  away  from  the  ideal  will  entail  risk.  The  science 
and  art  of  both  developmental  and  operational  testing  is  to  design  and  maintain  a  balance  where 
the  risks  are  acceptable. 

T&E  management  in  the  intricate  world  of  military  acquisition  is  one  of  the  most  complex  and 
politically  charged  jobs  one  can  imagine.  It  follows  that  the  range  and  depth  of  the  tool  kit 
required  by  the  test  community  to  accomplish  their  job  is  likewise  complex.  M&S  is  one  of 
those  tools.  Proper  application  of  M&S  is,  among  other  things,  an  integration  issue.  Whether 
the  top  level  T&E  plan  for  a  system  is  a  Test  and  Evaluation  Master  Plan  (TEMP)  or  a  Service 
T&E  plan,  it  must  reflect  the  balance. 
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Chapter  3 

STRUCTURED  APPROACH  FOR  THE  USE  OF  M&S 


The  structured  process  for  the  use  of  M&S  to  fill  gaps,  integrate  disparate  information,  cross 
check  test  results,  and  provide  balance  is  outlined  in  Table  3-1.  Specific  areas  where  shortfalls 
likely  will  occur  are  discussed  in  this  chapter.  The  selection  and  application  of  models  and 
simulations  must  be  in  concert  with  an  overall  test  and  evaluation  plan,  or  approach.  Sections 

3.1  through  3.9  provide  detailed  discussions  of  each  of  the  items  outlined  in  the  the  table. 

Table  3-1. 

OPERATIONAL  SUITABILITY  MODELING  AND 
SIMULATION  APPROACH 


•  Operational  Suitability  T&E  Planning 

1.  TEMP  Coverage 

2.  T&E  Planning 

3.  Model  Requirements  Analysis 

4.  Model  Selection  or  Development 

5.  Model  Verification  and  Validation  (V&V) 

6.  Model  Accreditation 

•  Phasing  the  M&S  Activity  With  the  Test  Activity 

•  Using  Comparability  Analysis  and  Other  Tools 

•  Using  T&E  Data  (DT&E,  as  well  as  OT&E) 

•  Crosschecking  and  Revalidation  of  Assumptions 

•  M&S  Results  as  an  Input  to  Focus  the  Testing 

•  Model  and  Data  Management 

•  Presentation  of  Data  to  Decisionmakers 

•  Understanding  and  Avoiding  M&S  Hazards 


3.1  Operational  Suitability  T&E  Planning 

M&S  requires  planning  and  integration  with  other  T&E  activities.  The  selection  of  the 
appropriate  model  or  modeling  environment,  and  the  verification  and  validation  (V&V)  of  the 
models,  are  critical  elements  of  this  plan.  Sections  3.1.1  through  3.1.5  detail  each  of  the 
planning  steps  that  are  shown  under  OS  T&E  planning  in  Table  3-1. 

3.1.1  Test  and  Evaluation  Master  Plan  (TEMP)  Coverage 

The  development  of  a  TEMP  is  part  of  the  top-level  process  for  designing  a  T&E  program  for 
major  systems.  Information  required  to  resolve  critical  issues  and  questions  should  be  identified. 
If  data  from  DT&E  and/or  OT&E  will  not  be  available  or  will  not  be  of  sufficient  detail  to  re¬ 
solve  those  issues  and  questions,  then  the  use  of  M&S  becomes  a  strong  candidate  for 
supplementing  the  actual  test  data. 
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If  the  use  of  M&S  is  anticipated  for  early  assessments  in  a  weapon  system  test  program,  then  it 
should  be  indicated  in  an  early  TEMP.  Defined  plans  for  the  use  of  M&S  should  be  presented  in 
later  TEMPs.  The  definition  of  M&S  activities  should  include  M&S  objectives.  Measures  of 
Effectiveness  (MOEs)  to  be  studied  or  modeled,  a  list  of  the  assumptions  inherent  in  the  effort, 
expected  inputs  to  the  M&S  effort,  expected  outputs,  a  phasing  plan  for  times  and  places  for  data 
availability,  the  amount  and  kinds  of  resources  allocated  to  the  M&S  effort,  and  a  list  of  models 
and  simulations  that  might  fit  the  requirement.  As  more  definition  is  developed,  the  models  to  be 
used  should  be  identified  and  plans  for  their  validation  described. 

3.1.2  Test  and  Evaluation  Planning 

Once  the  determination  to  use  M&S  has  been  made  and  the  information  detailed  in  section  3.1.1 
is  made  available,  an  M&S  plan  should  be  developed.  This  will  contain  an  expansion  of  the 
above  information,  as  well  as  a  more  detailed  plan  for  model  application.  Additional  details 
should  include  a  model  selection  plan,  a  model  verification  and  validation  (V&V)  plan,  a 
detailed  resource  requirements  analysis,  and  a  plan  for  integration  with  all  other  components  of 
the  top-level  T&E  plan. 

3.1.3  Model  Requirements  Analysis 

The  analysis  of  model  requirements  is  based  on  the  needs  identified  in  the  top-level  plan.  The 
requirements  are  further  refined  during  the  T&E  planning  phase,  and  documented  in  the  model 
selection  plan.  Some  of  the  key  factors  are  the  availability  and  stability  of  input  data,  the 
measures  of  effectiveness  of  interest,  the  granularity  (precision)  of  the  model,  special  hardware 
or  software  requirements,  cost  of  operation,  and  other  factors,  such  as  the  level  of  V&V  needed. 

3.1.4  Model  Selection  or  Development 

With  the  requirements  established  and  the  model  selection  plan  completed,  the  process  of 
reviewing  existing  models  and  assessing  the  time  and  level  of  effort  needed  to  develop  new 
models  is  an  accomplished  fact  Use  of  a  structured  requirements  approach  facilitates  efforts  to 
maintain  a  balance  in  this  area. 

3.1.5  Model  Verification  and  Validation  (V&V) 

T&E  results  are  being  increasingly  scrutinized  at  all  levels  within  and  outside  the  government. 
Credibility  of  M&S  evaluations  is  constantly  questioned  and,  as  a  result,  verification  and  valida¬ 
tion  of  the  respective  models  must  be  given  special  consideration  and  attention.  V&V  results 
must  pass  the  scrutiny  of  knowledgeable  and  informed  decisionmakers.  The  “DOT&E  Policy  for 
the  Application  of  Modeling/Simulation  in  Support  of  OT&E”  contains  the  following: 

Verification  is  the  process  of  determining  whether  a  computer  program  or  a 
simulation  model  performs  as  intended.  A  verification  plan  must  be  prepared  for  M/'S 
planned  for  use  in  operational  test  and  evaluation.  This  plan  must  be  referenced  in 
the  weapon  system  Test  and  Evaluation  Master  Plan  (TEMP).  For  new  and  modified 
simulation  models,  the  verification  plan  must  describe  the  verification  process(es)  and 
the  documentation  for  reporting  verification  results.  For  existing  simulation  models, 
previous  verification  efforts  which  led  to  accreditation,  if  any,  must  be  referenced. 
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Validation  is  the  process  which,  as  a  minimum,  addresses  the  following  primary 
concerns:  (1)  the  appropriateness  of  the  model  to  adequately  answer  the  questions  or 
issues  under  study;  (2)  the  degree  of  confidence  in  the  conclusions  that  can  be  drawn 
from  the  M/S  results;  (3)  the  appropriateness  of  the  threat  data  and  threat  tactics  used 
in  the  model;  and  (4)  model  consistency.  A  M/S  is  appropriate  if  it  addresses  the 
critical  issues  and  the  supporting  Measures  Of  Effectiveness  (MOEs),  and  if  the  M/S 
is  a  realistic  representation  of  the  weapon  system  and  its  operational  environment. 

M/S  appropriateness  depends  on  the  modeling  techniques,  assumptions  and 
limitations,  the  input  sources  and  quality,  the  ability  to  measure  performance,  and  the 
design  of  the  experiment.  Confidence  in  M/S  results  can  be  enhanced  by  comparison 
to  other  data,  e.g.,  actual  test  results,  other  models,  or  historical  data.  The  sensitivity 
(driving  and  limiting  factors)  should  be  well  understood  and  documented.  Plans  to 
recalibrate,  reverify,  and/or  revalidate  models  and  simulations  based  on  actual  test 
results  also  should  be  documented  and  implemented. 

3.1.6  Model  Accreditation 

The  “DOT&E  Policy  for  the  Application  of  Modeling/Simulation  in  Support  of  OT&E”  defines 

accreditation  as: 

“the  process  of  certifying  that  a  computer  model  has  achieved  an  established  standard 
such  that  it  can  be  applied  for  a  specific  purpose.” 


Accreditation  is  approval  by  management  -  based  on  experience  and  expert  judgment  -  that  a 
model  is  adequate  for  its  intended  use.  The  accreditation  mechanism  recognizes  that  V&V  of  a 
model  are  continual  processes  and  that  full  validation  of  the  model  may  not  be  technically  or 
economically  feasible.  Accreditation  is  possible  even  if  the  model  is  not  fully  validated,  but  it 
does  not  lessen  the  need  for  continuing  to  work  toward  full  V&V. 

3.2  Phasing  the  M&S  Activity  with  the  Test  Activity 

The  M&S  plan  must  be  coordinated  with  other  test  activities.  This  will  ensure  that  data 
availability  and  quality  issues  are  considered.  Other  phasing  issues  include  due  dates  for  reports, 
availability  of  test  assets,  test  schedules,  and  test  coverage. 

3.3  Using  Comparability  Analysis  and  Other  Tools 

At  times,  the  fidelity  of  test  assets  may  be  adequate  for  meeting  most  operational  effectiveness 
objectives  (i.e.,  brassboard,  breadboard,  prototype  being  tested  for  proof  of  concept),  but 
inadequate  for  testing  suitability  objectives.  In  many  cases,  specific  components  of  the  support 
structure  (e.g.,  support  equipment,  technical  data,  diagnostic  software,  etc.)  are  not  available  in  a 
timely  manner.  As  will  be  discussed  in  the  examples  in  Chapter  4,  there  are  tools  and 
techniques,  such  as  comparability  analysis,  that  have  been  applied  very  successfully  to 
supplement  actual  test  data.  These  techniques  permit  an  earlier  assessment  of  suitability  issues 
and  a  more  timely  identification  of  critical  design  features. 
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3.4  Using  T&E  Data  (DT&E,  as  well  as  OT&E) 


The  utility  and  sensitivity  of  DT&E  data  should  be  assessed.  In  many  cases,  the  increased 
dialogue  and  the  better  understanding  of  data  needs  by  the  two  communities  are  of  great  value 
not  only  for  M&S  use,  but  also  for  providing  additional  test  data  for  the  OT&E  evaluation 
process. 

3.5  Crosschecking  and  Revalidation  of  Assumptions 

In  any  analysis,  there  are  crosschecks  that  must  be  accomplished  to  ensure  consistency  and 
quality  results.  The  revalidation  of  the  assumptions  inherent  in  the  results  is  critical,  and  is  often 
overlooked  in  M&S.  It  is  good  practice  to  develop  an  illustration  for  use  in  repons  or  briefings 
that  clearly  documents  the  assumptions.  One  format  that  has  been  used  is  a  "balance  chan"  that 
lists  "Factors  and  Assumptions  That  Lead  to  Optimistic  Results"  on  the  left  and  "Factors  and 
Assumptions  That  Lead  to  Pessimistic  Results"  on  the  right.  This  type  of  illustration  provides  a 
beneficial  tool  to  the  user  of  the  M&S  results. 

3.6  M&S  Results  as  an  Input  to  Focus  the  Testing 

As  is  reponed  in  the  examples  in  Chapter  4,  a  powerful  result  of  M&S  is  providing  information 
that  can  significantly  improve  the  focus,  cost  effectiveness,  and  ultimate  value  of  traditional 
testing.  For  example,  early  assessment  of  the  operational  suitability  of  the  E-3A  identified  and 
helped  to  quantify  the  criticality  of  the  diagnostics  system.  This  assessment  assisted  in  the 
definition  of  diagnostics  MOEs  and  the  design  of  the  actual  testing. 

3.7  Model  and  Data  Management 

The  importance  of  this  area  cannot  be  overemphasized.  If  model  configuration  is  not  controlled 
and  V&V  is  lacking,  the  results  are,  at  best,  suspect  and,  at  worst,  disastrous.  Likewise,  the  input 
data  must  be  managed,  controlled,  and  verified.  Model  and  data  management  must  be  a  part  of 
the  M&S  plan,  with  proper  resources  identified  and  obtained. 

3.8  Presentation  of  Data  to  Decisionmakers 

The  best  of  models  and  simulations  and  their  results  are  of  little  value  if  the  presentation  of  those 
results  is  poorly  done.  It  is  not  as  important  that  a  specific  presentation  format  be  used  as  it  is 
that  the  presentation  be  designed  to  meet  specific  objectives.  There  first  must  be  agreement  on 
the  objectives.  If  possible,  they  should  be  in  writing.  The  key  is  to  present  the  data  and  the 
information  in  a  form  and  in  parameters  that  the  intended  audience  can  relate  to.  In  defense 
systems,  examples  of  parameters  that  are  important  to  operational  commanders  are  effective 
combat  sorties,  miles  steamed,  targets  destroyed,  probability  of  mission  success,  numbers  of 
systems  lost,  and  time  to  restore  to  mission-capable  status. 

3.9  Understanding  and  Avoiding  M&S  Hazards 

The  hazards  in  the  application  of  M&S  are  not  unlike  those  in  any  endeavor  —  rushing  results,  no 
time  for  V&V,  poor  or  incomplete  planning,  inadequate  funding,  untrained  or  inexperienced 
M&S  personnel,  results  that  very  few  may  want  to  hear,  "my  model  versus  your  model,"  junk 
data,  undocumented  assumptions,  lack  of  model  controls,  lack  of  data  controls,  "tinkering"  with 
model  results,  etc.  Most  of  these  are  common  problems.  The  best  defense  is,  again,  a  good 
offense.  Do  not  skimp  on  the  planning.  The  well  structured  M&S  plan  should  address  each  of 
these  potential  problem  areas. 
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Chapter  4 

EXAMPLES  OF  SUITABILITY  M&S 


This  chapter  presents  three  examples  of  the  application  of  M&S  to  operational  suitability  topics 
within  an  operational  test  and  evaluation  environment.  These  examples,  all  drawn  from  OT&E, 
are  included  to  show  the  breadth  of  M&S  applications  and  to  demonstrate  the  effectiveness  of 
some  of  the  methods  employed  in  providing  information  to  decisionmakers. 

4.1  The  F-16  Aircraft 

The  example  of  the  F-16  fighter  is  one  of  the  best  representations  of  a  continuous  suitability 
modeling  and  simulation  effort  throughout  the  acquisition  phases  of  a  system.  This  M&S  effort 
began  during  the  competition  between  the  F-16  and  the  F-17  in  the  early  1970's  and  continued 
through  the  end  of  the  FOT&E  in  the  early  1980's.  The  simulation  model  continued  to  be 
updated  and  used  for  maintenance  manpower  analysis  even  after  the  test  program  was 
completed.  The  primary  model  used  was  LCOM  (Logistics  Composite  Model),  although  other 
models,  using  QGERT  and  its  successors,  SLAM  I  and  n,  also  were  developed  and  applied. 
This  coordinated  M&S  effort  addressed  a  wide  spectrum  of  application  areas,  as  is  reflected  in 
the  following: 

•  In  order  to  develop  models  of  the  two  competing  designs  during  the  flyoff,  comparability 
analysis  was  used  to  both  develop  representations  of  each  aircraft  design  and  to  compare 
the  two  designs.  Two  segregated  teams  developed  the  models  of  each  aircraft.  The 
teams  worked  with  the  respective  contractors  and  program  office  personnel,  as  well  as 
using  command  personnel  in  developing  the  models. 

•  Sortie  generation,  availability,  maintainability,  manpower,  configuration  baseline,  and 
other  decision  criteria  were  analyzed.  One  key  area  of  the  analysis  was  the  comparison 
of  the  one-engine  design  of  the  F-16  with  the  two-engine  design  of  the  F-17. 

•  During  both  IOT&E  and  FOT&E,  an  LCOM-developed  model  was  applied  to  the 
analysis  of  specific  deficiencies  identified  during  operational  testing.  Well  over  200 
deficiencies  were  analyzed.  The  primary  question  for  each  deficiency  was  "what  is  the 
real  impact?"  This  M&S  activity  assisted  in  answering  the  impact  question  for  those 
deficiencies  that  existed  in  the  operational  suitability  arena.  The  results  were  very 
effective  and  were  used  by  the  decisionmakers.  This  process  was  named  QED  (Quantita¬ 
tive  Evaluation  of  Deficiencies).  It  was  lauded  by  the  Commander  of  the  using 
command,  the  OT&E  agency,  the  government  program  manager,  and  the  developing 
contractors. 

•  The  models  also  were  used  during  the  entire  acquisition  period  to  develop  manpower 
projections  and  to  determine  the  manpower  standard. 

•  A  companion  model  to  LCOM,  TAC  FLIER,  was  used  to  analyze  and  estimate  the 
aircrew  ratio  as  part  of  the  process  to  determine  the  required  aircrew  manpower. 

This  broad-based  M&S  effort  spanned  nearly  ten  years.  Even  today,  it  can  be  considered  as  a 
standard  for  effective  application  of  M&S.  The  models  were  always  considered  "tools"  for 
decisionmakers  and  served  as  "communications  baselines"  among  users,  the  program  office,  the 
contractor,  and  developmental  test  and  operational  test  personnel.  The  models  were  useful  in 
refining  requirements  and  promoting  a  better  understanding  of  the  operational  scenario  and  its 
impact. 
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4.2  The  E-3A  AWACS 


The  application  of  operational  suitability  M&S  to  the  E-3A  AWACS  is  another  example  of  an 
early  application  of  M&S  that  continued  throughout  the  acquisition  phases.  M&S  was  used  from 
the  "brassboard"  prototype  of  the  E-3A  in  the  mid-1970's  to  well  after  the  FOT&E  in  the  early 
1980's.  As  with  the  F-16  example,  M&S  was  applied  in  a  variety  of  areas.  Analysis  of  processes 
such  as  maintenance,  integrated  logistics  support  issues,  operational  scenario  impacts, 
requirements  analysis,  and  training  were  accomplished.  Some  specific  M&S  activities  included 
the  following: 

•  Early  requirements  and  requests  for  decision  information  led  to  some  spreadsheet-type 
analysis  using  the  scenario  information  and  the  data  that  were  being  assembled  for 
building  the  more  detailed  modeling  tools.  Due  to  the  overriding  impact  (open-the-door 
costs)  of  dispersion  of  the  aircraft  to  geographically  separated  operating  locations,  results 
from  very  high  level  (macro)  models  were  especially  accurate  and  useful  very  early  in 
the  program. 

•  The  flow  of  maintenance  for  the  mission  avionics  was  especially  critical,  due  to  the 
heavy  reliance  on  the  diagnostics  system.  Models  were  used  to  evaluate  alternative  plans 
for  maintenance  "work  arounds"  for  subsystems  where  the  diagnostics  fell  short.  That 
work  led  to  further  analysis  and  identification  of  additional  training  requirements  and  the 
need  for  "beyond  BIT"  (built-in  test)  technical  data;  that  is,  when  the  BIT  no  longer  can 
aid  in  troubleshooting,  what  procedures  should  be  followed? 

•  One  special  aspect  of  the  M&S  effort  was  the  heavy  involvement  of  the  Air  Force  system 
support  manager  from  Oklahoma  City  Air  Logistics  Center.  Many  special  investigations 
of  the  spare  parts  requirements  and  depot  activities  were  carried  out  because  of  this 
involvement. 

•  Later  in  the  program,  fully  developed  models,  using  LCOM,  were  used  (as  in  the  F-16 
example)  to  analyze  the  impact  of  deficiencies  and  to  design  unique  support  techniques 
to  achieve  maximum  results  with  limited  resources. 

4.3  Intermediate  Level  Test  Stations 

In  this  example,  an  early  analysis  determined  the  quantity  of  maintenance  test  stations  that  were 
needed  for  each  operational  unit.  This  analysis  was  flawed.  The  use  of  a  steady  state  analysis 
resulted  in  an  estimate  of  a  test  station  quantity  that  could  not  support  the  wartime  sortie  rate 
planned  for  the  aircraft. 

•  The  use  of  a  squadron-level  simulation  model  using  the  LCOM  included  modeling  the 
intermediate-level,  automatic-test-station  environment.  The  model  used  realistic 
reliability  and  maintainability  values  for  the  system  components  and  the  planned  system 
utilization  to  ascertain  if  the  test  station  complement  could  satisfy  the  wartime  sortie 
requirement.  The  simulation  indicated  that  additional  stations  were  required. 

•  The  earlier  assessment  of  the  true  situation  permitted  the  program  manager  to  obtain 
additional  funding  and  to  increase  the  number  of  test  stations  provided  to  the  operational 
units. 
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Chapter  5 
SUMMARY 


The  importance  of  Operational  Suitability  is  not  in  question!  Suitability  related  elements  of  de¬ 
fense  systems  are  critical  to  mission  performance  and  easily  account  for  more  than  half  of 
weapon  system  costs. 

There  are  both  direct  and  indirect  areas  of  suitability  cost.  The  direct  areas  include  such  suitabil¬ 
ity  elements  as  manpower,  spare  parts,  support  equipment,  technical  data,  training,  sustaining 
engineering,  and  software  maintenance.  The  indirect  areas  are  those  in  which  some  effects  of 
operational  suitability  elements  show  up  in  the  aggregate.  These  indirect  areas  are  especially 
well  suited  to  M&S  because  even  with  "full-up,"  "hands-on"  testing,  the  aggregate  effects  of 
small  problems  are  virtually  impossible  to  see,  and  the  need  for  additional  manpower  or  other 
support,  due  to  shortfalls  in  suitability  areas  such  as  reliability,  maintainability,  or  support 
equipment,  can  be  quite  dramatic.  Other  indirect  effects  such  as  downtime,  lost  missions, 
ineffective  missions,  and  lost  or  destroyed  equipment  or  personnel  also  can  be  effectively 
addressed  through  M&S. 

The  payoffs  from  M&S  activity  are  high  leverage,  and  offer  very  high  benefit-to-cost  ratios. 
Some  of  these  payoffs  include: 

•  Early  identification  of  issues  and  critical  areas. 

•  Identification  of  key  objectives  to  better  focus  other  phases  of  test  and  evaluation. 

•  Assessment  of  the  sensitivity  and  impact  of  key  design  areas,  such  as  diagnostics. 

•  Improved  communications  on  important  factors,  such  as  the  planned  operational 
and  support  scenarios. 

•  The  ability  to  provide  impact  assessments  that  are  critical  to  the  effectiveness  and 
ultimate  utility  of  T&E  to  the  decisionmaker. 

The  T&E  community  faces  some  exceptional  challenges  in  the  future.  To  meet  these  challenges, 
operational  suitability  M&S  is  a  powerful  tool  to  add  to  the  T&E  tool  kit.  It  has  been  applied 
very  successfully  in  the  past  and  can  be  applied  even  more  effectively  in  the  future.  It  also  can 
be  misused  and  abused.  The  better  it  is  understood  and  the  more  that  is  learned  from  experience, 
the  better  prepared  we  will  be  to  profit  from  the  application  of  operational  suitability  modeling 
and  simulation. 
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Appendix  A 

M&S  SYSTEMS,  ENVIRONMENTS,  AND  MODELS 


Introduction 

This  Appendix  presents  some  examples  of  simulation  and  modeling  tools  that  may  be  considered 
for  use  in  the  performance  of  Operational  Suitability  (OS)  analysis.  The  examples  presented 
consist  of  tools  currently  in  use  or  in  the  final  stages  of  development  or  testing.  Each  is  de¬ 
scribed  using  a  standard  format,  including  basic  model  data,  point  of  contact,  and  system 
requirements  (hardware/software).  The  models  and  systems  described  are  classified  into  one  of 
the  following  three  categories: 

Automated  Systems:  An  automated  system  is  a  deterministic  model  that  lacks  random 
events.  It  may  be  a  manual  process  that  has  been  automated  to  save  time.  This  category 
also  includes  spreadsheet  applications. 

Simulation  Environment:  A  simulation  environment  provides  the  analyst  with  a  system  to 
create  models  when  the  need  for  a  specific  application  exists  and  a  model  has  not  been  de¬ 
veloped.  The  analyst  is  provided  only  the  necessary  tools  to  develop  the  model.  The  envi¬ 
ronment  has  a  language  (i.e.,  SLAM  II  or  SEMPLE_1)  that  may  be  used  to  construct  a  simu¬ 
lation  model.  It  also  incorporates  the  functions  necessary  to  simulate  the  passage  of  time 
(timing  routine),  to  track  events,  and  to  gather  statistics  on  entities,  events,  and  queues  dur¬ 
ing  the  simulation. 

Simulation  Model:  A  simulation  model  comprises  an  application  that  models  random 
events,  simulates  the  passage  of  time,  and  performs  data  collection.  The  model  is  designed 
to  perform  in  a  standard  way,  providing  consistent  (although  not  necessarily  the  same)  data 
from  one  run  to  the  next. 

Examples  of  the  defense  systems  for  which  each  model  was  applied  are  provided.  The  following 
categorization  of  systems  is  used: 

Ships:  Any  surface  or  submarine  vessel. 

Land  Vehicles:  Any  system  designed  to  operate  on  land,  including  tanks,  trucks,  etc. 

Air  Vehicles:  Any  vehicle  designed  to  operate  above  the  earth’s  surface,  but  remain  within 
the  earth’s  atmosphere.  This  includes  all  conventional  aircraft  (fixed-wing  and  rotary  wing). 

Space:  Any  system,  vehicle,  or  platform  designed  for  and  used  in  space,  including  low- 
earth-orbit  vehicles. 

Field  Equipment:  Any  piece  of  powered  or  non-powered  equipment  needed  to  support  a 
weapon  system  or  operational  system.  Field  equipment  includes  ground  power  units,  air 
conditioners,  and  field  test  sets. 

Command  and  Control  (C&C):  Any  system  or  platform  used  to  perform  command  and 
control  functions,  including  airborne  radar  platforms,  ground  radars,  and  the  communica¬ 
tions  facilities  required  to  operate  the  command  and  control  systems. 

Missiles:  Any  air-to-air  or  air-to-ground  weapon,  including  defensive  and  offensive  weap¬ 
ons,  excluding  ground-to-air  defensive  weaponry. 

Air  Defense:  All  weapons  designed  to  destroy  or  counter  airborne  threats,  including  rapid 
fire  cannons  (i.e.,  Phalanx  CIWS)  and  surface-to-air  missiles  (i.e.,  PATRIOT).  This  catego¬ 
ry  does  not  include  the  command  and  control  function. 
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The  efforts  used  to  verify  and  validate  the  example  systems  also  are  described.  In  addition,  a 
brief  overview  is  presented,  followed  by  an  illustration  that  depicts  the  utility  of  the  model  to 
address  the  operational  suitability  areas  of  availability,  sustainability,  reliability,  maintainability, 
provisioning,  and  manpower. 

The  dark  shading  on  the  illustration  represents  the  area(s)  of  primary  application  of  the  model; 
light  shading  is  used  to  reflect  secondary  areas  that  offer  the  greatest  potential  for  additional  ap¬ 
plication  of  the  model’s  capability.  The  numbers  in  the  squares  refer  the  reader  to  the  notes 
below  the  chart  which  provide  further  discussion. 

In  the  case  of  the  summary  charts  (page  A-37),  the  numbers  in  the  squares  show  which  models 
are  applicable  for  each  of  the  respective  areas.  The  survey  summary  is  provided  (pages  A-36, 
A-37)  to  illustrate  the  total  coverage  for  the  various  models.  Shading  is  used  to  depict  the 
applicable  areas  of  coverage,  presenting  the  reader  with  a  broad  picture  of  the  capabilities 
offered  by  surveyed  systems  and  models. 

This  Appendix  consists  of  the  following  examples: 

Automated  Systems: 

1.  Aviation  Readiness  Requirements  Oriented  to  WRAs  (ARROWS) 

2.  Weapon  System  Reliability  Model  (WSRM) 

3.  Spreadsheet  Models 

Simulation  Environments: 

4.  Modeler’s  Workbench 

5.  Simulation  Language  for  Alternative  Modeling  (SLAM  II) 

6.  SIMPLE_1 

Simulation  Models: 

7.  TRASANA  Aircraft  Reliability  and  Maintainability  Simulation  (TARMS  II) 

8.  Combat  Analysis  Sustainability  Model  (CASMO) 

9.  Logistics  Composite  Model  (LCOM) 

10.  TIGER 

11.  Comprehensive  Aircraft  Support  Effectiveness  Evaluation  (CASEE) 

12.  Comprehensive  Operational  Support  Evaluation  Model  for  Space  (COSEMS) 

Simulation  Survey  Summary 
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Model  Applications:  This  model  has  been  applied  to  a  wide  variety  of  Naval  aircraft  and 
support  equipment  for  the  purpose  of  defining  budget  requirements.  It  presently  is  being  applied 
to  the  V-22  program. 

Verification:  The  verification  process  must  include  a  thorough  review  to  ensure  the  ARROWS 
Model  will  perform  as  designed.  When  the  ARROWS  Model  is  used  as  part  of  OT&E,  results  of 
this  verification  effort  must  be  documented  and  included  in  the  Test  and  Evaluation  Master  Plan 
(TEMP). 

Validation:  Model  validation  was  accomplished  by  the  Fleet  Material  Support  facility  in  1986. 
The  test  consisted  of  data  obtained  from  the  1986  deployment  of  the  USS  Enterprise  and  its 
contingent  of  F-14  aircraft.  Data  on  701  organizational  maintenance  items  were  collected  and 
entered  into  the  3M  standard  Navy  data  system.  These  data  were  used  with  the  ARROWS 
Model  to  make  predictions  of  parts  failures,  parts  availability,  repair  times,  and  spare-based 
requirements.  The  ARROWS  Model  output  was  compared  to  actual  operational  data,  with 
results  being  very  close. 

Purpose;  ARROW  is  a  stochastic,  analytical  model  used  to  determine  aviation  spares 
requirements  to  support  the  Navy  consumer-level  spares  program.  It  may  use  the  Navy  standard 
database  for  input  data,  or  data  provided  by  the  user.  The  primary  inputs  are  reliability  and 
maintainability  data  for  each  part  under  consideration. 
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Application  To  Operational  Suitability 
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1.  Provisioning:  The  user  inputs  R&M  data  for  each  part,  and  the  model  predicts  the  spares 
required.  It  also  can  be  used  to  determine  spares  requirements  for  shore-based  aircraft  or 
shipboard  aircraft  maintenance. 
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Formerly  named  -  Mission  Critical  Success 


Model  Category: 


Automated  System 


Point  Of  Contact: 


Mr.  Neal  Chamblee,  USAF,  AFOTEC 


■BD3BI 


Commercial  (505)  846-1  264  |  Autovon  |  246-1264 


VAX  11/780 


Coded  Irt: 


FORTRAN 


Model  Applications:  This  model  has  been  used  successfully  for  the  B-1B,  F-15E,  and  F-16 
programs. 

Verification:  The  model  has  been  tested  to  ensure  that  the  code  is  error  free,  and  that  it  per¬ 
forms  as  designed.  Output  results  were  compared  with  previous  manual  calculations  to  complete 
the  verification  effort. 

Validation:  WSRM  undergoes  a  validation  process  with  each  application.  The  AFOTEC 
simulation  committee  monitors  the  application  of  each  model,  including  WSRM.  Once  the 
model  is  configured  for  a  particular  system,  it  is  run  using  field  data  from  a  similar  system.  The 
results  of  the  trial  runs  are  compared  to  actual  field  data  for  the  similar  system  to  ensure  the 
model  is  producing  accurate  results. 

Purpose:  The  Mission  Critical  Success  Probability  (MCSP)  Model  has  been  renamed  the 

Weapon  System  Reliability  Model  (WSRM).  It  is  a  deterministic  model  residing  on  the 
AFOTEC  VAX  computer  system.  A  microcomputer  version  of  the  model  is  under  development. 
The  user  inputs  factors  such  as  mission  length,  minimum  essential  equipment  list,  and  hardware 
failure  rates.  The  model  then  generates  the  expected  reliability  data  that  subsequently  will  be 
used  in  one  of  the  simulation  models  used  by  AFOTEC  (LCOM  or  a  specific  model  developed  in 
SLAM  II).  This  model  is  best  described  as  a  preprocessor  for  simulation  models. 
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1.  Reliability:  The  model  is  used  to  determine  expected  system  reliability  given  specific 
hardware  failure  rates  and  mission  characteristics.  Data  from  the  model  are  used  as  inputs  to 
simulation  models.  All  events  are  deterministic.  Randomness  is  not  a  model  feature. 
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Model  Applications:  Spreadsheet  models  have  been  used  extensively  in  OT&E  of  numerous 
weapon  systems,  including  the  U.S.  Army  DIVAD  (Sergeant  York)  and  the  U.S.  Air  Force  F-16. 

Verification:  The  user  must  verify  that  the  spreadsheet  model  performs  as  expected.  If  it  is 
intended  to  use  the  model  as  part  of  the  OT&E  process,  a  verification  plan  must  be  included  in 
the  Test  and  Evaluation  Master  Plan  (TEMP). 

Validation:  The  user  must  ensure  that  the  spreadsheet  model  is  appropriate  for  the  intended 
task.  The  results  of  the  model  must  be  consistent  and  defendable.  Quality  input  data  and  careful 
experiment  design  will  improve  confidence  in  the  model.  Results  should  be  compared  with  test 
data  or  actual  historical  data  as  an  additional  validity  check. 

Purpose:  Spreadsheet  models  are  constructed  using  various  software  packages,  including  Lotus 
1-2-3,  Multiplan,  and  Supercalc.  These  models  normally  are  created  and  used  on  micro¬ 
computers.  The  main  objective  of  spreadsheet  analysis  is  to  eliminate  the  need  for  manual 
calculations.  The  benefit  derived  from  this  application  is  the  ability  to  conduct  “what-if  ’  analy¬ 
sis  involving  complex  calculations  and  formulas.  As  the  user  inputs  changes  to  the  spreadsheet 
cell  calculations,  the  entire  spreadsheet  reflects  the  latest  changes.  Once  formulas  are  entered 
into  the  appropriate  cells,  the  analyst  is  free  to  enter  revised  data  for  analysis.  Many  commercial 
spreadsheet  software  packages  have  built-in  graphics  functions  to  depict  the  data  on  X  and  Y, 
Bar,  or  Pie  charts. 

Using  spreadsheet  models,  the  user  can  specify  cell  location  and  data  for  a  formula.  This 
capability  is  extended  in  some  available  software  to  allow  the  user  to  specify  external 
spreadsheets.  Most  spreadsheet  packages  permit  the  user  to  specify  data  inputs  from  other 
computer  files  and  to  direct  output  to  a  number  of  locations  including  a  spreadsheet  file,  printer, 
computer  screen,  or  a  data  file. 

Spreadsheets  provide  user  control  and  are  very  flexible  and  useful  for  early  analysis  where  data 
are  limited.  Once  the  spreadsheet  model  is  created,  the  user  can  input  data  as  they  become  avail¬ 
able,  using  estimates  for  unknowns  until  the  actual  data  are  collected. 
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Modeler’s  Workbench 

Model  Category:|  Simulation  Environment 


Point  Of  Contact:  |Mr.  Doug  Williams,  Advanced  Technology,  Inc. 


Commercial 

(205)  895-0396 

Autovon 

N/A  | 

Hosted  On:|  Symbolics 


Coded  ln:|  Common  LISP 


Model  Applications:  The  model’s  initial  application  was  on  studies  of  the  Advanced  Launch 
System  (ALSYM),  a  joint  NAS  A/US  AF  program. 

Verification:  The  Modeler's  Workbench  has  undergone  extensive  reviews  including  peer  code 
reviews,  as  well  as  unit,  integration  and  production  level  testing  to  ensure  the  model  performs  as 
intended.  When  used  in  OT&E,  the  verification  efforts  must  be  documented  and  referenced  in 
the  Test  and  Evaluation  Master  Plan  (TEMP). 

Validation:  The  Modeler's  Workbench  has  not  undergone  validation  as  defined  in  3.1.5  (see 
page  3-2).  This  simulation  environment  is  designed  to  model  space  systems  that  are  presently  in 
the  concept  definition  phase;  therefore,  comparison  of  predictions  to  actual  system  performance 
has  not  been  possible.  When  these  systems  become  functional,  the  model  will  undergo  formal 
validation. 

Purpose:  The  Modeler's  Workbench  is  a  set  of  programming  tools  for  the  development  of 
knowledge-based  models  and  simulations.  It  was  developed  on  a  Symbolics  computer  using 
Common  LISP  and  the  Flavors  package  as  part  of  a  comprehensive  analysis  capability. 

The  Modeler's  Workbench  comprises  three  major  components:  an  object-oriented  modeling 
environment,  a  rule-based  inference  package,  and  a  discrete  event  simulation  package.  The 
modeling  environment  and  simulation  packages  are  designed  to  provide  the  user  with  an 
environment  customized  for  the  rapid  prototyping  of  models  and  the  simulations  defined  on 
those  models.  Once  an  approach  has  been  selected,  it  has  the  flexibility  to  enable  the  prototype 
to  evolve  into  a  working  simulation.  In  addition,  a  rule-based  inference  facility  is  incorporated 
to  provide  flexibility  in  developing  a  knowledge-based  approach,  as  well  as  a  procedural 
approach,  to  decisionmaking.  Together,  these  features  encourage  and  enhance  an  incremental 
development  methodology  to  problem  solving. 

The  initial  application  of  the  Workbench  was  to  simulate  operations  of  the  Advanced  Launch 
System  (ALSYM).  The  ALSYM  is  a  joint  NASA/USAF  program  to  develop  an  alternate  launch 
vehicle  for  lifting  heavy  payloads  into  space.  The  primary  area  of  interest  to  the  modeler  is  the 
ALSYM  logistics  support  infrastructure  with  emphasis  on  vehicle  assembly,  payload  integration, 
and  the  refurbishment  of  reusable  components.  The  infrastructure  simulation  is  integrated  with  a 
reliability  prediction  model,  a  cost  model,  and  a  global  evaluation  model  to  provide  a 
comprehensive  assessment  of  resources  needed  to  support  the  Advanced  Launch  System. 
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The  Modeler's  Workbench  currently  is  being  used  to  develop  a  simulation  of  the  operation  of  the 
space  station  “Freedom.”  The  primary  area  of  investigation  is  the  space  station  operational 
availability.  The  Modeler's  Workbench  permits  analysis  of  tradeoffs  among  various  design 
factors  such  as  reliability  and  maintainability,  and  management  factors  such  as  the  space  station 
logistics  support  structure.  Results  of  these  analyses  will  reveal  the  impact  of  alternative 
combinations  of  these  factors  for  decisionmakers  to  use  in  program  management. 

The  Modeler’s  Workbench  is  used  primarily  for  assessing  reliability,  maintainability,  and  avail¬ 
ability  characteristics  of  a  system.  In  addition,  a  cost  module  provides  costing  per  pound  to 
orbit,  as  well  as  total  life  cycle  costing  information. 

Application  To  Operational  Suitability 


MODEL 

OBJECTIVES 

SHIPS 

LAND 

VEH 

AIR 

VEH 

m 

FIELD 

EQUIP 

C&C 

MISSILES 

AIR 

DEF 

AVAILABILITY 

SUSTAINABILITY 

RELIABILITY 

MAINTAINABILITY 

PROVISIONING 

MANPOWER 

llli 

■mm 

PI 

m 

S&Z 

■ 

KWi 

yyy.y 

■ 

LEGEND:  H  =  PRIMARY  APPLICATION  |~J  =  SECONDARY  APPLICATION 


1.  Availability:  Applications  developed  with  the  Modeler’s  Workbench  may  be  used  to 
determine  the  operational  availability  of  a  space-based  system  (space  station)  or  a  launch  vehicle. 

2.  Reliability:  The  Modeler's  Workbench  permits  tradeoff  studies  to  determine  the  reliability 
of  the  system  to  perform,  given  the  available  logistics  support  infrastructure. 

3.  Maintainability:  Tradeoff  studies  may  be  used  to  optimize  maintenance  and  logistic  support 
structures,  thus  maximizing  the  systems  operational  availability. 
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Simulation  Language  for  Alternative  Modeling  (SLAM  It) 


Model  Category: 


Simulation  Environment 


Point  Of  Contact: 


Lt.  Col.  Kunkle,  USAF,  AFOTEC/LG4  * 


Commercial  (505)  844-0348  Autovon  244-0348 


Hosted  On: 


VAX 


Coded  In: 


SLAM  II 


Model  Applications:  SLAM  has  been  used  by  AFOTEC  to  develop  models  for  a  variety  of 
weapon  systems,  including  the  F-15E  and  F-16  aircraft  and  the  SRAM  missile. 

Verification:  The  user  must  assure  that  the  model  developed  using  SLAM  II  performs  as 
intended.  This  may  consist  of  peer  code  review  and  examination  of  the  network  logic.  AFOTEC 
instituted  an  internal  model  review  committee  to  accomplish  these  verification  tasks.  The 
verification  plan  must  be  included  in  the  Test  and  Evaluation  Master  Plan  (TEMP)  for  all  models 
intended  for  use  in  OT&E. 

Validation:  SLAM  If  is  a  simulation  environment;  therefore,  validation  must  be  an  integral  pan 
of  the  process  for  building  each  model.  The  intended  use  of  the  SLAM  II  Model  must  be  exam¬ 
ined  to  ensure  that  it  is  suitable  for  the  application.  The  model  is  considered  appropriate  if  it 
addresses  the  critical  issues  and  the  supporting  Measures  Of  Effectiveness  (MOEs),  and  if  it  is  a 
realistic  representation  of  the  weapon  system  and  its  operational  environment.  Validation  plans 
and  previous  validation  efforts  must  be  documented  and  referenced  in  the  TEMP.  The  degree  of 
confidence  in  the  model  varies  with  the  quality  of  input  data,  modeling  techniques,  and  the 
experiment  design.  Confidence  in  the  SLAM  II  Model  may  be  enhanced  by  comparison  of  the 
output  results  to  test  data,  other  models,  or  historical  data. 

Purcflse;  SLAM  II  is  a  simulation  environment.  AFOTEC  uses  this  model  development 
environment  to  develop  weapon-system-specific  models.  SLAM  II  uses  FORTRAN  for  the 
development  of  special  function  subroutines  and  each  model  developed  by  AFOTEC  makes 
extensive  use  of  FORTRAN.  These  models  are  designed  specifically  to  answer  the  questions 
encountered  during  operational  test;  they  are  unique  to  the  weapon  system  for  which  they  are 
developed.  The  user  inputs  standard  scenario  data,  maintenance  process  flows,  and  maintenance 
resource  availability.  SLAM  II  models  are  useful  during  operational  suitability  assessments 
since  each  model  can  be  specifically  tailored  to  address  the  issues  determined  to  be  critical  for 
the  specific  weapon  system. 

The  use  of  SLAM  II  modeling  techniques  permits  flexibility  not  available  in  traditional  models. 
Careful  consideration  must  be  given  to  model  accuracy  with  a  clear  and  traceable  audit  trail  for 
all  model  development  decisions. 


♦NOTE:  Questions  on  uses  of  SLAM  II,  beyond  those  at  AFOTEC,  should  be  directed  to  Ms.  Sue  Knoop,  (317)  879-1011. 
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1.  Availability:  The  analyst  may  constrain  resources  and  determine  the  impact  on  a  weapon 
system's  sortie  rate. 

2.  Sustainability:  The  simulation  can  be  used  to  assess  the  ability  of  a  unit  to  sustain 
operations,  given  constrained  resources.  One  example  of  a  possible  application  of  a  model 
written  in  SLAM  II  is  "How  long  can  the  unit  sustain  combat  operations  with  a  limited  set  of 
spare  parts,  as  in  the  War  Reserve  Spares  Kit  (WRSK)?"  The  level  of  resolution  could  be 
downsized  to  component  level  rather  than  unit  level  (i.e.,  the  ability  to  keep  the  radar  functioning 
with  limited  repair  resources). 

3.  Maintainability:  The  user  can  obtain  maintainability  results  through  constructions  of  the 
SLAM  networks. 

4.  Manpower:  The  model  can  be  used  to  estimate  manpower  requirements  by  Air  Force 
Specialty  Code  (AFSC). 
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SIMPLE  1 


Mode!  Category: 


Simulation  Environment 


Point  Of  Contact:  1  Mr.  Phil  Cobbln,  Sierra  Simulations  &  Software 


Commercial  I  (800)  446-3697  |  Autovon 


N/A 


Hosted  On:|  IBM  PC,  XT,  AT,  &  True  Compatible  Microcomputers 


SIMPLE  1 


Model  Applications:  SIMPLE_1  is  used  primarily  in  the  academic  and  civilian  sectors; 
however,  some  manpower  models  have  been  developed  for  the  U.S.  Navy. 

Verification:  The  user  must  assure  that  the  model  developed  using  SIMPLE_1  performs  as 
intended.  This  may  consist  of  peer  code  review  and  examination  of  the  network  logic.  The 
verification  plan  must  be  included  in  the  Test  and  Evaluation  Master  Plan  (TEMP)  for  all  models 
intended  for  use  in  OT&E. 

Validation:  SIMPLE_1  is  a  simulation  environment;  therefore,  validation  must  be  included  in 
each  model  building  process.  The  intended  use  of  the  SIMPLE_1  Model  must  examined  in 
detail.  The  model  is  considered  appropriate  if  it  addresses  the  critical  issues  and  the  supporting 
Measures  Of  Effectiveness  (MOEs),  and  if  it  is  a  realistic  representation  of  the  weapon  system 
and  its  operational  environment.  Validation  plans  and  previous  validation  efforts  must  be 
documented  and  referenced  in  the  TEMP.  The  degree  of  confidence  in  the  model  varies  with  the 
quality  of  input  data,  modeling  techniques,  and  the  experiment  design.  Confidence  in  the 
SIMPLE_1  Model  may  be  enhanced  by  comparison  of  the  output  results  to  test  data,  other 
models,  or  historical  data. 

Purpose:  SIMPLE_1  provides  the  capability  to  develop  simulation  models  as  well  as  pre-  and 
post-processors  used  to  support  the  simulation  project.  In  addition,  the  SIMPLE_1  environment 
permits  the  user  to  develop  utility  programs  that  may  be  used  for  such  tasks  as  data  collection  or 
reduction.  Once  models  are  developed,  the  user  may  compile  the  model  using  the  RUNSIM  op¬ 
tion.  The  model  may  then  be  re-hosted  as  a  stand-alone  executable  system. 

SIMPLE_1  provides  an  array  of  graphics  utilities  that  permit  the  modeler  to  animate  the  simula¬ 
tion  and  present  a  BIT-mapped  display  via  EGA  or  VGA  output  devices. 

The  model  supports  continuous  as  well  as  discrete  event  simulation  capability.  SIMPLE_1  pro¬ 
vides  extensive  statistics  collection  capability  for  events  that  occur  during  simulation.  The  user 
may  declare  variables  and  statistics  requirements,  perform  file  input/output  operations,  and 
animate  the  simulation  results  in  real  time. 

The  modeler  may  establish  multiple  run  requirements  or  use  SIMPLE_1  features  to  determine 
multiple  run  parameters  based  on  the  model  behavior. 
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Discrete  systems  consist  of  networks  that  define  the  flow  of  events  or  entities  within  the  model. 
Entities  may  be  grouped  and  flow  through  the  network  while  still  retaining  their  unique  at¬ 
tributes.  In  addition,  dissimilar  entities  may  be  assembled  to  form  a  group.  The  entities  flow 
through  the  blocks  in  the  network  and  may  represent  objects  such  as  tools,  parts,  and  people. 

Continuous  models  are  constructed  through  the  use  of  differential  equations.  An  example  of 
continuous  model  methodology  is  the  change  in  thrust-to-weight  ratio  as  a  rocket  consumes  fuel 
following  launch.  SIMPLE_1  provides  the  capability  to  integrate  discrete  event  simulation  and 
continuous  simulation  through  die  use  of  built-in  system  functions. 

Application  To  Operational  Suitability 
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1.  Availability:  The  analyst  may  constrain  resources  and  determine  the  impact  on  the  weapon 
system's  sortie  rate. 

2.  Sustainability:  The  simulation  also  can  be  used  to  assess  the  ability  of  a  unit  to  sustain 
operations,  given  constrained  resources. 

3.  Maintainability:  The  user  can  obtain  maintainability  results  through  constructions  of  the 
SIMPLE_1  networks. 

4.  Manpower:  The  model  can  be  used  to  estimate  manpower  requirements. 
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TRASANA  Aircraft  Reliability  and  Maintainability 
_ CAA  Version  II  (TAR MS  W _ _ 


Modal  Category:! 


Simulation  Model 


[  Point  Of  Contact:  |  Dr.  Dong  Kim,  U.S.  Army,  Concepts  Analysis  Agency  1 


Telephone:)  Commercial  |  (301 )  29 5 -2088  |  Autovon  |  295-2088  | 


Hosted  Omf 


VAX,  Unisys 


1 


|  Coded  in :1 


Simscript  11.5 


Model  Applications:  TARMS  II  has  been  applied  to  all  Army  aircraft,  both  fixed-wing  and 
rotary  wing. 

Verification:  TARMS  II  has  undergone  extensive  reviews,  including  peer  code  reviews  and 
desk  checking,  to  ensure  the  model  performs  as  intended.  When  used  in  OT&E,  the  verification 
process  must  be  documented  and  referenced  in  the  Test  and  Evaluation  Master  Plan  (TEMP). 

Validation:  TARMS  II  has  not  undergone  a  formal  validation  procedure.  The  model  is 
presently  undergoing  extensive  modifications,  and  will  undergo  formal  validation  procedures 
once  modifications  are  completed. 

Purpose:  TARMS  II  is  a  stochastic,  process-oriented,  simulation  model  which  represents  the 
operation  of  any  U.S.  Army  aviation  organization,  from  company  to  theater  in  size.  The  model 
may  simulate  both  peacetime  and  wartime  scenarios,  driven  by  a  detailed  list  of  mission 
requirements  for  the  duration  of  the  game.  The  Blue  force  is  modeled  as  a  set  of  aircraft  of 
various  types  organized  into  company-sized  elements,  each  element  having  its  own  maintenance 
personnel,  test  equipment,  and  stock  of  repair  parts.  The  Red  force  is  modeled  as  a  mix  of 
weapon  systems,  each  representing  a  threat  to  the  Blue  force.  While  performing  a  mission,  a  mix 
of  Blue  aircraft  may  experience  mechanical  failures  and  combat  damage  that  may  result  in  a  total 
loss  of  the  aircraft,  a  forced  landing  in  the  field,  an  abort  of  the  mission,  or  the  generation  of  a 
required  maintenance  action.  Combat  may  be  two-sided,  with  both  Red  and  Blue  exchanging 
rounds.  However,  where  a  hit  on  Blue  results  in  attrition  or  damage,  the  result  of  Blue  engaging 
Red  is  either  a  miss  or  a  kill.  Some  assumptions  associated  with  TARMS  II  include  the  follow¬ 
ing:  Ground  vehicles  used  to  transport  contact  teams  do  not  experience  failures  or  combat 
damage  and  are  assumed  to  be  available  at  all  times;  fuel  and  ammunition  are  assumed  available 
at  all  times;  each  aviation  organization  may  use  only  one  aviation  intermediate  maintenance 
(AVIM)  facility,  thus  the  corps  AVIM  slice  cannot  explicitly  be  modeled  at  division  level; 
recovery  kits  are  assumed  available  at  all  times;  cannibalization  of  aircraft  destroyed  in  the  field 
is  not  modeled;  contact  teams  work  on  only  one  aircraft,  and  then  return  to  base;  each  Blue 
aircraft  is  limited  to  one  on-board  weapon  type;  each  hit  of  a  Blue  aircraft  by  a  Red  weapon  is  a 
unique  event;  and  the  cumulative  effect  of  multiple  hits  is  not  modeled. 
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1.  Availability:  The  user  has  the  option  of  constraining  resources  to  determine  its  effect  on  the 
availability  of  the  system. 

2.  Sustainability:  The  model  considers  various  levels  of  maintenance  and  may  be  used  to 
analyze  the  ability  of  a  unit  to  perform  required  missions,  given  equipment  failures  and  the 
capability  of  supply  to  furnish  replacement  pans.  The  failed  parts  are  modeled  through  the 
“back”  repair  shops  and  through  depot-level  repair. 

3.  Reliability:  The  user  may  input  various  reliability  factors  to  determine  the  impact  on  the 
unit’s  ability  to  meet  mission  requirements. 

4.  Maintainability:  The  model  may  be  used  to  analyze  maintenance  concepts  such  as 
deferrals,  use  of  contact  teams  to  fix  forward,  repair  part  order  and  ship  time,  and  controlled 
substitution  policies.  The  support  system  requirements,  such  as  Military  Occupational  Speciality 
(MOS)  staffing  levels,  quantity  of  test  equipment,  repair  parts  acquisition  and  stockage  policies, 
number  of  “float”  aircraft,  and  the  number  and  location  of  aircraft  for  recovery  and  contact  team 
missions  also  may  be  modeled. 

5.  Manpower:  The  model  may  be  used  to  make  initial  determinations  of  manpower 
authorizations  required  to  support  a  given  scenario.  The  authorizations  are  by  MOS,  and  are 
used  for  requirements  estimates  only. 
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Combat  Analysis  Sustainability  Modal  {CAS MO) 


|  Model  Category:!  Simulation  Model 

|  Point  Of  Contacts  J  Dr.  Dong  Kim,  U.S.  Army,  Concepts  Analysis  Agency 
1  Telephoned  Commercial  |  (301 )  295-2088  |  Autovon  |  295-2088 
|  Hosted  Qftj|  VAX,  Unisys 

lllllllilll  Simscript  11.5 


Model  Applications:  The  model  currently  is  in  development. 

Verification:  CASMO  currently  is  undergoing  extensive  reviews,  including  peer  code  reviews 
and  desk  checking,  to  ensure  the  model  performs  as  intended.  When  used  in  OT&E,  the 
verification  plan  must  be  documented  and  included  in  the  Test  and  Evaluation  Master  Plan. 

Validation:  A  new  model,  CASMO  currently  is  in  the  process  of  being  delivered  by  the 
contractor  and  has  not  undergone  formal  validation.  Once  the  complete  model  is  delivered  to  the 
government,  it  will  undergo  a  validation  process. 

Purpose:  CASMO  will  rely  on  data  produced  by  the  Force  Concepts  Evaluation  Model 
(FORCEM)  and  Vector-In-Commander  (VIC)  Model.  The  model  will  have  preprocessors  to 
perform  the  necessary  data  translation.  It  will  model  ground  activity,  but  will  be  somewhat 
similar  to  TARMS  II  (see  page  A-22).  Unit  combat  activity  data  are  required  by  CASMO, 
FORCEM,  and  VIC  to  produce  the  combat  scenario  driver.  CASMO  uses  preprocessors  to 
perform  the  necessary  conversion  on  the  outputs  of  FORCEM  and  VIC.  The  user  must  manually 
gather  and  enter  data  (e.g.,  organizational  structure  and  the  number  of  each  type  weapon  system). 
Functionally,  CASMO  will  model  attrition,  movement.  Command,  Control,  and 
Communications,  and  logistics  for  ground  forces.  The  scenario  preprocessor  decomposes  the 
combat  model  unit  (division  for  FORCEM  or  battalion  for  VIC)  into  company-sized  combat 
arms  units  (armor,  infantry,  artillery)  and  positions  the  company-sized  units,  according  to 
doctrine,  within  the  division  area.  The  company-sized  unit  is  assigned  its  authorized  quantity  of 
weapon  systems.  The  preprocessor  also  positions  division-level  maintenance  and  supply 
organizations  within  the  division  area. 

CASMO  uses  shotline  data  contained  in  Sustainability  Predictions  for  Army  spare  components 
Requirements  for  Combat  (SPARC)  similar  to  the  TARMS  II  procedure.  These  data  predict  the 
specific  damage  expected  from  each  hit  of  an  enemy  weapon.  Shotline  data  are  produced  by  the 
Ballistics  Research  Laboratory  (BRL).  CASMO  uses  the  preprocessor  from  TARMS  II  to 
process  shotline  data  for  its  use.  As  with  TARMS  II,  the  shotline  preprocessor  produces  a  series 
of  files,  one  for  each  Red-on-Blue,  killer-victim  combination.  Each  file  contains  a  random 
selection  of  several  thousand  shotlines;  each  shotline  contains  data  pertaining  to  a  hit  at  a 
specific  location  and  aspect  angle.  The  data  also  contain  the  list  of  parts,  and  the  probability  that 
that  part  is  damaged.  In  addition,  the  resources  required  to  repair  or  replace  the  component 
(damage  repair)  are  contained  in  the  file. 
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CASMO  deterministically  computes  scheduled  maintenance  for  each  weapon  system  within  each 
unit.  Unscheduled  maintenance  is  determined  stochastically  for  each  type  of  weapon  system. 
The  model  also  determines  the  expected  number  of  failures  resulting  from  the  specified  aging 
and  the  user-input  system  failure  rates.  For  each  failure,  a  specific  part  is  randomly  selected, 
based  on  the  user-input  relative  part-failure  rate.  A  specific  maintenance  action  (e.g.,  remove 
and  replace)  is  then  selected,  based  on  the  probabilities  specified  by  the  user.  Combat  damage 
also  is  modeled,  using  scenario  specified  hit  data.  CASMO  selects  a  shotline,  which  gives  the 
probability  of  each  part  being  hit.  The  model  stochastically  selects  the  parts  hit,  and  then  sets  the 
corresponding  maintenance  actions  in  a  manner  that  is  analogous  to  unscheduled  maintenance. 

Application  To  Operational  Suitability 


1.  Availability:  The  analyst  can  constrain  various  resources  to  determine  the  impact  on 
weapon  system  availability. 

2.  Sustainability:  CASMO  is  an  event-step  simulation  model  for  the  operation  of  the  logistics 
support  base  within  Army  divisions.  It  emphasizes  the  sustainability  of  major  ground-based 
weapons  systems  in  an  operational  environment  and  assesses  the  ability  of  major  weapon 
systems  to  meet  mission  requirements.  It  also  provides  data  on  the  division-level  supply  and 
maintenance  organization  and  policy  to  determine  their  effect  on  weapon  system  performance. 

3.  Reliability:  The  model  allows  the  examination  of  weapon  system  failure  characteristics. 
The  system  failure  rate  is  given  in  hours,  rounds  fired,  and  miles.  The  user  inputs  the  relative 
frequency  of  component  failure  and  the  implied  assignment  of  parts  to  one  of  the  three 
subsystems  (mobility,  firepower,  other).  The  user  also  specifies  the  system  redundancy,  thus 
adjusting  the  probability  of  parts  failure  and  the  probability  of  the  resulting  maintenance  action. 

4.  Maintainability:  The  model  can  be  used  to  compute  scheduled  maintenance  and  determine 
unscheduled  maintenance  and  combat  damage. 

5.  Provisioning:  Repair  parts  assets  can  be  evaluated  with  the  user  establishing  the  manner  in 
which  repair  parts  inventory  is  modeled.  Method  1  establishes  an  authorized  quantity  at  each 
type  maintenance  unit.  Method  2  assigns  a  probability  of  fill  at  each  type  maintenance  unit. 

6.  Manpower:  The  model  may  be  used  to  make  an  initial  determination  of  manpower 
authorizations,  consisting  of  quantities  and  skill  levels,  required  to  support  a  given  scenario. 


Appendix  A  -  Simulation  Models 


Page  A-25 


Logistics  Composite  Model  (LOOM) 


Model  Category: 


Simulation  Model 


Point  Of  Contact: 


Lt.  Col.  Kunkle,  AFOTEC/LG4* 


1 

Commercial 

(505)  844-0348 

Autovon  | 

Hosted  On: 


Honeywell,  CDC,  IBM,  CRAY,  VAX 


Coded  In: 


Slmscript  11.5,  FORTRAN,  COBOL  (Pre/Post  Processors) 


and  E-3A  programs. 


The  model  has  been  used  in  the  B-1B,  F-15,  P-16,  F-20,  F/EF-111A/D, 


Verification:  LCOM  has  undergone  extensive  reviews,  including  peer  code  reviews  and  desk 
checking,  to  ensure  the  model  performs  as  intended.  When  used  in  OT&E,  the  verification 
efforts  must  be  documented  and  referenced  in  the  Test  and  Evaluation  Master  Plan  (TEMP). 

Validation:  Initial  validation  took  place  at  Seymour-Johnson  AFB,  SC.  This  test  involved  the 
deployment  of  a  squadron  of  F-4  aircraft  to  a  remote  section  of  the  ramp,  and  all  resources  were 
constrained  to  levels  used  within  the  LCOM  simulation.  The  deployed  aircraft  used  the  same 
flying  scenario  as  that  which  was  used  in  the  simulation.  The  simulation  projections  matched 
almost  exactly  the  performance  demonstrated  by  the  deployed  aircraft.  In  more  recent  examples, 
AFOTEC  has  validated  the  model  by  comparing  results  with  field  data  from  comparable 
systems.  This  technique  was  used  to  validate  F-15  and  F-16  simulation  models. 

Purpose:  To  obtain  the  necessary  versatility  of  being  able  to  apply  LCOM  to  a  wide  range  of 
weapon  systems,  the  majority  of  the  simulation  logic  is  provided  externally  in  data  rather  than  as 
pan  of  the  software  itself.  Users  provide  data  in  the  form  of  networks  that  represent  the  system 
being  modeled.  These  externally  provided  data  constitute  roughly  75%  of  the  decision  logic. 

During  the  simulation  process,  when  a  mission  is  requested  an  attempt  is  made  to  complete  it  by 
obtaining  the  needed  aircraft  from  a  ready  pool  and  beginning  their  processing  through  a  set  of 
user-defined  pre-sonie  tasks.  A  take-off  and  cancellation  time  also  is  established  by  the  mission 
request.  Once  the  required  aircraft  complete  the  set  of  pre-sortie  tasks  and  the  take-off  time  is 
reached,  they  begin  the  sortie  task  (fly  the  sortie).  Upon  completion  of  this  task,  they  must 
undergo  the  defined  post-sortie  task.  When  completed,  the  aircraft  are  returned  to  the 
serviceable  pool.  The  time  to  cancel  will  be  utilized  only  if  insufficient  aircraft  are  ready  to  fly 
the  mission,  or  if  the  pre-sortie  task  cannot  be  completed  in  the  allotted  time. 

The  pre-sortie  and  post-sortie  blocks  are  user-supplied  logic  networks  that  represent  sections  of 
logic  for  the  main  module  to  process.  Pre-sortie  networks  could  include  tasks  such  as  preflight 
checks  and  weapons  loading.  Post-sortie  tasks  might  include  fuel  service,  post-flight  inspection, 
unscheduled  and  scheduled  maintenance,  etc.  Tasks  within  these  networks  will  define  the 
resources  required  to  do  the  task  priority  of  work  relative  to  other  defined  maintenance  and  their 
relationship  to  succeeding  tasks.  The  details  of  the  task  networks  are  user  controlled  and  limited 
only  by  the  availability  of  data  and  the  computer  memory  required  to  process  the  task  networks. 

*NOTE:  Questions  on  uses  of  LCOM,  beyond  those  at  AFOTEC,  should  be  directed  to  Msgt.  Allan  Bishop,  (512)  652-2833. 
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The  main  module  of  the  LCOM  software  is  designed  to  simulate  a  broad  range  of  aircraft 
operations,  scheduling,  maintenance,  and  supply  functions  at  an  Air  Force  Base.  In  accordance 
with  the  LCOM  design  objectives,  the  logical  ordering  of  actions  within  the  simulation  may  be 
adapted  to  many  problem  situations  and  therefore  must  be  specified  by  the  user  for  his  specific 
application.  The  user  is  free  to  define  both  the  resources  of  interest  and  the  manner  of  their 
intended  utilization,  while  the  software  provides  the  necessary  controls  and  structure  to  simulate 
and  maintain  a  record  of  their  action  and  interaction. 

Operations  data  include  the  number  of  missions,  mission  type,  time  of  take-off,  sortie  duration, 
number  of  aircraft  required,  and  other  related  mission  data,  included  in  the  data  are  the  activities 
and  user-defined  external  requirements  for  either  aircraft  or  non-aircraft  resources  that  are  to  be 
processed  in  such  a  manner  that  does  not  lead  to  or  involve  a  sortie  task.  Maintenance  data 
utilized  are  primarily  AFM  66-1/AFM  66-5  data  in  a  form  of  a  maintenance  network.  These  data 
include  necessary  scheduled  maintenance  actions  required  on  both  aircraft  and  non-aircraft 
resources.  Supply  data  include  supply  demand  and  resupply  processes  at  the  various  levels  of 
part  indentures,  i.e.,  assembly,  sub-assembly,  sub-sub-assembly,  module,  etc.,  according  to  the 
user  definition.  Generally,  Work  Unit  Code  references  are  used. 

Application  To  Operational  Suitability 


MODEL 

OBJECTIVES 

SHIPS 

LAND 

VEH 

Pi 

SPACE 

FIELD 

EQUIP 

c&c 

MISSILES 

AIR 

DEF 

AVAILABILITY 

liyll 

U 

m 

bh 

SUSTAINABILITY 

■ 

■  V:-  •. 

2 

2 

2 

RELIABILITY 

■ 

■ 

■ 

MAINTAINABILITY 

■ 

3 

3 

3 

PROVISIONING 

■ 

'•  -  . 

■ 

MANPOWER 

■ 

'•.•yji'y  ///St. 

wmm. 

5 

■ 

LEGEND:  M  =  PRIMARY  APPLICATION  EH  =  SECONDARY  APPLICATION 


1.  Availability:  The  model  can  be  used  effectively  to  determine  availability  of  aircraft  or  other 
resources  during  simulation  experiments. 

2.  Sustainability:  Sustainability  assessments  may  be  produced  using  LCOM.  Spares  or  other 
resources  can  be  constrained,  and  the  associated  effect  on  the  ability  of  the  unit  to  fly  the 
required  number  of  sorties  can  be  examined. 

3.  Maintainability:  The  model’s  flexible  network  architecture  provides  the  ability  to  perform 
maintainability  analysis  at  whatever  maintenance  level  is  appropriate. 

4.  Provisioning:  LCOM  could  be  used  for  determination  of  parts  requirements,  but 
traditionally  this  function  has  been  handled  by  other  Air  Force  models  (i.e.,  DYNA-METRIC). 

5.  Manpower:  One  of  the  primary  uses  for  LCOM  within  the  Air  Force  is  the  determination 
of  aircraft  maintenance  manpower  requirements.  This  methodology  has  been  very  successful, 
and  its  use  in  this  application  has  been  validated  and  accepted. 
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Model  Category:) 

Simulation  Model  | 

(  Point  Of  Contact:  | 

Mr.  Palmer  Luetjen,  NAVSEA,  SEA05MR  | 

Telephone:) Commercial  (202)  692-21 50  i Autovon  222-2150  | 

|  Heated  On:) 

Various  computer  systems  f 

|  Coded  ln:| 

FORTRAN  | 

Model  Applications:  Applications  of  the  TIGER  Model  include  every  ship  in  the  inventory. 

Verification:  TIGER  has  undergone  extensive  reviews,  including  peer  code  reviews  and  desk 
checking,  to  ensure  the  model  performs  as  intended.  When  used  in  OT&E,  the  verification 
efforts  must  be  documented  and  referenced  in  the  Test  and  Evaluation  Master  Plan  (TEMP). 

Validation:  One  of  the  initial  validation  efforts  took  place  during  builders’  trials  at  Puget  Sound 
in  1970.  TIGER  was  used  to  predict  the  reliability  of  the  Patrol  Hydrofoil  (PHM)  and  the  results 
were  compared  to  the  reliability  measured  during  the  trials.  Cases  where  the  predicted  reliability 
failed  to  match  the  actual  reliability  were  traced  to  errors  in  data  entry  or  data  reporting.  Both 
the  contractor  (Boeing)  and  NAVSEA  participated  in  the  validation  effort.  Prior  to  using  TIGER 
on  any  new  system,  the  mission  profile  must  be  approved  to  assure  it  is  representative  of  the 
intended  profile.  A  deterministic  model,  REX,  is  used  to  predict  the  reliability  for  one  or  more  of 
the  less  complex  subsystems.  Manual  calculations  also  may  be  accomplished  using  Markov 
Chain  theory.  TIGER  then  is  used  to  predict  reliability  for  the  same  subsystem.  The  results  are 
compared  to  ensure  TIGER  is  providing  reasonable  reliability  estimates  for  the  particular  system. 

Purpose:  TIGER  is  an  event-driven,  stochastic  simulation  model.  It  can  predict  the  reliability, 
maintainability,  and  availability  for  all  systems,  ranging  from  the  smallest  subcomponent  to  total 
ship  or  fleet  of  ships.  It  is  a  large  and  versatile  simulation  model,  especially  created  for  the  study 
of  the  Reliability,  Maintainability,  and  Availability  (RMA)  characteristics  of  systems.  The 
model  can  accept  data  from  standard  databases  (i.e.,  Navy  3M  Database).  Standard  TIGER  input 
information  consists  of  mission,  equipment,  and  system  configurations. 

The  user  must  input  the  weapon  system  objectives  and  functional  requirements.  This  starts  with 
the  mission  profile,  which  contains  a  full  description  of  mission  objectives  as  well  as  the  system 
and  equipment  functions  required  to  achieve  these  objectives.  During  the  mission  profile 
development  and  throughout  the  system  and  equipment  design  process,  more  detailed 
information  must  be  derived  to  support  the  engineering  analyses  that  are  part  of  the  application 
of  TIGER.  The  mission  profile  data  must  be  translated  into  a  timeline  format  acceptable  to  the 
TIGER  computer  program.  The  levei  of  detail  provided  to  the  TIGER  model  by  the  timelines  is 
determined  by  the  extent  of  information  provided  in  the  engagement  database. 

Reliability  Block  Diagram  methodology  represents  the  breakdown  of  the  hardware  system  being 
analyzed  into  its  major  items  and  the  interfaces  among  these  items.  Each  block  may  represent 
either  an  equipment  function  or  a  hardware  element.  The  next  step  in  equipment  definition  is  to 
establish  the  equipment  parameters.  These  factors  are  quantitative  measurements  for  the 
Reliability  and  Maintainability  (R&M)  and  the  duty  cycle  of  each  equipment. 


Page  A- 28 


Appendix  A  -  Simulation  Models 


The  user  must  clearly  specify  all  system  operating  rules.  This  set  of  data  describes  the  effects  of 
equipment  failures  on  system  and  mission  success,  considering  the  capabilities  of  the  system  to 
repair  equipment  failures.  This  description  also  must  discuss  the  maintenance  and  spares  support 
policy.  TIGER  also  can  represent  the  spares  support  stocking,  issuing,  and  reordering  of  spares. 
The  model  is  flexible  and  will  either  allow  unlimited  spares,  or  apply  spares  constraints. 

The  internal  model  operation  can  have  one  of  three  internal  equipment  events:  failure  of  an 
equipment,  arrival  of  a  spare  for  awaiting  equipment,  or  repair  of  an  equipment.  When  TIGER 
initiates  a  mission,  all  equipment  items  are  in  good  condition  and  full  logistic  support  is 
available.  The  user  has  the  ability  to  alter  the  initial  conditions  by  calling  special  preprocessors 
that  track  and  accumulate  usage  time,  along  with  the  associated  failures  and  spares  consumption. 

TIGER  repons  the  results  of  the  simulation  in  terms  of  system  and  equipment  indexes.  These 
estimators  of  system  performance  include  system  RMA,  measures  of  system  activity  during  the 
mission,  estimates  of  long  term  stable  system  characteristics,  standard  deviation,  equipment 
performance  statistics,  and  critical  equipment  assessment.  The  primary  figures  of  merit  are 
reliability,  instantaneous  availability,  critical  equipment  lists,  and  average  availability.  A  sec¬ 
ondary  purpose  of  TIGER  is  to  provide  sensitivity  analysis  on  systems  under  evaluation. 

Application  To  Operational  Suitability 


MODEL 

OBJECTIVES 

LAND 

VEH 

AIR 

VEH 

SPACE 

FIELD 

EQUIP 

C&C 

MISSILES 

AIR 

DEF 

AVAILABILITY 

■>//////; 

■ 

1 

SUSTAINABILITY 

'///■"/.'/////S'. 

■ 

■ 

2 

RELIABILITY 

wmm> 

vyyyy/yyy. 

■ 

3 

MAINTAINABILITY 

&■§£  ||  v  •' 

■ 

PROVISIONING 

■ 

■ 

■1 

MANPOWER 

■ 

■ 

■ 

LEGEND:  EM  =  PRIMARY  APPLICATION  dU  =  SECONDARY  APPLICATION 


1.  Availability:  An  index  of  performance  calculated  by  TIGER  is  instantaneous  availability.  It 
is  defined  as  the  probability  that  the  system  will  be  "up"  (be  capable  of  operating  satisfactorily) 
at  a  stated  time.  The  instantaneous  availability  is  calculated  at  the  beginning  and  end  of  each 
phase.  TIGER  also  calculates  average  availability,  i.e.,  the  probability  that  a  system  is  up  and 
capable  of  satisfactory  operation  at  any  (random)  point  in  the  timeline. 

2.  Sustainability:  This  model  can  determine  the  ability  to  sustain  an  engagement,  given 
equipment  failures  and  spares  replacement. 

3.  Reliability:  An  advantage  of  TIGER  over  similar  "mission  reliability  models"  (MIL-STD- 
756B)  is  that  the  TIGER  format  allows  direct  utilization  of  the  basic  building  blocks  of  reliability 
engineering,  the  Reliability  Block  Diagram  (RBD). 

4.  Maintainability:  This  model  provides  estimators  of  system  performance,  including  system 
maintainability  measures,  during  the  mission. 
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Model  Applications:  The  model  has  been  applied  to  virtually  all  naval  tactical  aircraft, 
including  helicopters  and  anti-submarine  aircraft. 

Verification:  CASEE  has  undergone  extensive  reviews,  including  peer  code  reviews  and  desk 
checking,  to  ensure  the  model  performs  as  intended.  When  used  in  OT&E,  the  verification 
efforts  must  be  documented  and  referenced  in  the  Test  and  Evaluation  Master  Plan  (TEMP). 

Validation:  An  example  of  CASEE  validation  took  place  in  1985,  during  the  first  carrier 
deployment  of  the  F-18  aircraft.  The  test  involved  two  squadrons  of  F-18  aircraft  (20  aircraft) 
involved  in  a  standard  Navy  carrier  operational  scenario.  The  test  was  conducted  by  members  of 
the  F-18  Assistant  Program  Manager  for  Logistics  (APML)  program  office  and  a  support 
contractor.  Information  Spectrum,  Inc.  Predictions  derived  from  CASEE  were  compared  with  F- 
18  operational  data  available  in  the  3M  database.  Predictions  for  both  Reliability  and 
Maintainability  (R&M)  and  readiness  were  good,  and  the  test  was  considered  successful.  Since 
the  1985  test,  the  model  has  undergone  several  further  validation  tests  involving  the  comparison 
of  model  predictions  to  actual  operational  data  for  carrier-based  aircraft. 

Purpose:  Model  application  traditionally  starts  early  in  the  acquisition  process  (at  or  prior  to 
Milestone  0).  CASEE  is  used  primarily  for  aircraft  operations  simulation.  The  aircraft  can  be 
ship-  or  shore-based  and  may  be  fixed-  or  rotary  wing.  The  enhancements  include  preprocessors 
that  simplify  data  acquisition,  drawing  data  from  LSAR  and  3M  databases  and  converting  them 
into  a  format  acceptable  to  CASEE.  Postprocessors  provide  decision  support  tools  and  enhanced 
model  output,  to  include  graphs  and  detailed  day-by-day  reports. 

CASEE  has  been  linked  to  the  REPAIRS  Model  that  provides  assessment  of  the  depot  repair 
actions  and  spares  pipeline  flow.  Prior  to  interface  with  REPAIRS,  depot  and  pipeline  delays 
were  calculated  as  a  wait  time.  This  procedure  was  less  than  accurate,  because  pipeline  delays 
are  dynamic  and  do  not  demonstrate  a  linear  relationship.  The  addition  of  the  depot  model  has 
provided  an  additional  resolution  of  detail  for  the  analyst. 

CASEE  uses  an  aggregate  approach  to  look  at  individual  work  centers.  It  looks  at  work  backlog 
by  work  center  rather  than  by  individual  maintenance  task. 

The  analyst  has  the  ability  to  use  simulation  at  Milestone  0  to  perform  suitability  analysis.  Once 
requirements  are  specified,  CASEE  can  be  used  to  simulate  the  system  to  determine  its  ability  to 
perform  satisfactorily.  This  early  analysis  may  depend  on  comparability  analysis  techniques 
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when  system  data  are  not  available.  Such  early  assessment  can  predict  availability,  given  R&M 
factors  for  various  components. 
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1.  Availability:  This  model  determines  an  aircraft’s  ability  to  meet  mission  requirements  by 
examination  of  the  complete  aircraft  system  and  aircraft  multiple  missions.  It  also  may  be  used 
to  determine  availability  in  terms  of  essential  equipment  or  systems.  It  develops  a  ratio  of 
“mission-capable”  to  “total  time.” 

2.  Sustainability:  This  model  determines  mission  capability,  a  key  measure,  and  provides 
inputs  for  readiness  and  resources  studies.  It  also  generates  realistic  cost  factors  for  evaluation, 
aids  in  defining  military  capability,  and  considers  battle  damage  and  attrition. 

3.  Reliability:  The  model  can  be  used  to  determine  the  impact  of  improvements  in  reliability  in 
terms  of  availability  and  manpower  savings. 

4.  Maintainability:  Information  on  maintenance  tasks,  maintenance  manpower,  and  queues 
are  generated  by  this  model. 

5.  Manpower:  Output  reports  may  be  analyzed  to  determine  manpower  requirements  to  meet 
objectives  in  a  given  scenario.  This  model  also  provides  data  on  the  maintenance  manpower 
utilization. 
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Model  Applications:  The  COSEMS  is  being  applied  to  the  Strategic  Defense  System  (SDS) 
satellites  and  satellite  constellations. 

Verification:  COSEMS  has  undergone  extensive  reviews  including  peer  code  reviews,  as  well 
as  unit,  integration  and  production  level  testing  to  ensure  the  model  performs  as  intended.  The 
model  also  was  tested  by  an  outside  independent  consultant,  and  the  independent  derivation  of 
the  algorithms  was  verified  by  another  contractor.  When  used  in  OT&E,  the  verification  efforts 
must  be  documented  and  referenced  in  the  Test  and  Evaluation  Master  Plan  (TEMP). 

Validation:  COSEMS  has  not  undergone  validation  as  defined  in  3.1.5  (see  page  3-2).  This 
simulation  environment  is  designed  to  model  space  systems  that  are  presently  in  the  concept 
definition  phase;  therefore,  comparison  of  predictions  to  actual  system  performance  has  not  been 
possible.  When  these  systems  (specifically,  the  space  station)  become  functional,  the  model  will 
undergo  formal  validation. 

Purpose:  The  COSEMS  Model  has  been  developed  to  investigate  the  support  alternatives  for 
satellites  and  satellite  constellations.  The  current  approach  for  maintaining  satellite  availability 
is  to  replace  entire  satellites  when  they  fail,  either  by  launching  a  replacement  satellite  or 
activating  an  in-orbit  spare.  Efforts  to  reduce  the  costs  to  support  satellites  in  large  constella¬ 
tions,  such  as  those  proposed  for  the  Strategic  Defense  System,  has  led  to  the  consideration  of  al¬ 
ternative  support  concepts  such  as  those  based  on  in-orbit  support.  This  calls  for  building  satel¬ 
lites  with  standardized  modules  —  orbital  replacement  units  or  (ORUs)  —  that  can  be  replaced  in 
orbit  using  tele-robotic  devices  attached  to  an  orbital  maneuvering  vehicle  (OMV).  To  evaluate 
these  support  concepts,  COSEMS,  a  large-scale  simulation  of  ground  and  space  operations,  has 
been  developed.  The  model  simulates  the  complex  dynamic  interactions  among  the  elements  of 
the  SDS,  its  support  system,  and  the  expendable  launch  delivery  system.  The  model  is  of  suffi¬ 
cient  detail  to  provide  realistic  quantitative  assessment  of  in-orbit  support  alternatives  relative  to 
the  approach  of  unit  satellite  replacement.  The  model  permits  the  prediction  of  (a) 
constellation/ring/payload  availability  versus  time,  (b)  the  impact  of  system  reliability,  maintain¬ 
ability,  and  supportability  on  availability,  and  (c)  the  development  of  initial  conditions  for  other 
modeling  and  simulation  efforts,  including  wargaming  and  engagement  models. 
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1.  Availability:  Operational  availability  of  the  satellite  constellation,  orbital  ring,  and  payload 
are  primary  outputs  of  the  model.  User-specified  minimum  operational  availability  data  are  used 
as  inputs  to  the  decision  to  implement  the  primary  or  backup  response,  when  this  is  applicable. 

2.  Sustainability:  The  ability  of  a  satellite  constellation  to  perform  its  intended  mission  may  be 
analyzed,  given  hardware  failures,  consumable  depletion  events,  delays  in  transporting  satellites, 
ORUs,  consumables  and  other  equipment  into  space,  and  delays  in  performing  unmanned  main¬ 
tenance  and  service  in  space. 

3.  Reliability:  System  block  diagrams,  level  of  redundancy  in  each  subsystem,  and  shape  and 
scale  parameters  in  Weibull  models  of  each  satellite  module  are  developed  by  the  user  (during  an 
interactive  session  with  the  preprocessor). 

4.  Maintainability:  The  user  may  analyze  maintainability  considering  support  concept,  sup¬ 
port  system  architecture,  constellation  architecture,  spacecraft  design  (extent  of  replaceable  mod¬ 
ules,  module  replacement  time),  and  other  in-orbit  operations  times  (e.g.,  OMV  payload  integra¬ 
tion  time). 

5.  Provisioning:  The  model  predicts  the  annual  number  of  ORUs  that  will  be  required  to  main¬ 
tain  a  satellite  constellation  through  its  life  cycle. 


Appendix  A  -  Simulation  Models 


Page  A-33 


THIS  PAGE 
INTENTIONALLY 
LEFT  BLANK 


Page  A-  34 


Appendix  A  -  Simulation  Models 


SURVEY  SUMMARY 


Appendix  A  -  Survey  Summary 


Page  A-35 


Model  Category: 


AH  Categories 


Point  Of  Contacts!  See  Respective  Individual  Listing 


|  Telephone;!  Commercial 


Ref.  Above 


Autovon  |  Ref.  Above  f 


Uliosted  On:|  See  Respective  Individual  Listing 


|  Coded  ln:| 


See  Respective  Individual  Listing 


Summary:  The  automated  systems,  simulation  environments,  and  simulation  models  included 
in  this  document  arc  summarized  on  the  following  page.  The  name  and  associated  number  for 
each  model  or  system  reviewed  and  the  page  number  that  contains  the  detailed  description  of  the 
model  are  listed  below.  A  summary  of  the  suitability  attributes  of  the  models  (1  -  12)  are  shown 
in  the  figures  on  the  facing  page.  The  top  figure  represents  the  primary  model  or  system 
applications;  the  bottom  figure  represents  the  secondary,  or  potential,  areas  of  application.  These 
figures  may  be  used  to  determine  the  appropriate  model  or  system  for  a  particular  application. 
By  referring  to  the  page  number  of  the  selected  model  (see  listing  below),  the  reader  can  obtain  a 
more  detailed  description  of  the  model  or  system.  The  spreadsheet  (see  automated  systems,  item 
3  below)  was  considered  so  generic  that  the  utility  of  presenting  it  in  the  summary  chart  was  neg¬ 
ligible.  If  a  model  or  system  is  not  available  for  a  particular  function,  and  calculations  are  re¬ 
quired  during  the  analysis,  the  use  of  a  spreadsheet  should  be  considered. 

The  subsections  within  this  Appendix  consist  of  the  following: 

Page 


Automated  Systems: .  A-5 

1 .  Aviation  Readiness  Requirements  Oriented 

to  WRAs  (ARROWS) .  A-6 

2.  Weapon  System  Reliability  Model  (WSRM) .  A-8 

3.  Spreadsheet  Models .  A- 10 

Simulation  Environments: .  A-13 

4.  Modeler’s  Workbench .  A-14 

5.  Simulation  Language  for  Alternative  Modeling  (SLAM  II) .  A- 16 

6.  SIMPLE.  1 .  A- 18 

Simulation  Models: .  A-21 

7.  TRASANA  Aircraft  Reliability  and  Maintainability 

Simulation  (TARMS  II) .  A-22 

8.  Combat  Analysis  Sustainability  Model  (CASMO) .  A-24 

9.  Logistics  Composite  Model  (LCOM) .  A-26 

10.  TIGER .  A-28 

11.  Comprehensive  Aircraft  Support  Effectiveness 

Evaluation  (CASEE) .  A-30 

12.  Comprehensive  Operational  Support  Evaluation 

Model  for  Space  (COSEMS) .  A-32 
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Figure  A-2.  SUMMARY  OF  SECONDARY  OBJECTIVES 
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